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MEMORANDUM  FOR  ASSISTANT  SECRETARY  OF  DEFENSE  (COMMAND, 

CONTROL,  COMMUNICATIONS,  AND 
INTELLIGENCE) 

DIRECTOR,  DEFENSE  PROCUREMENT 
DIRECTOR,  DEFENSE  LOGISTICS  AGENCY 


SUBJECT:  Allegations  to  the  Defense  Hotline  Concerning  the  Standard  Procurement 
System  (Report  No.  96-219) 


We  are  providing  this  audit  report  for  your  review  and  comment.  We 
performed  the  audit  in  response  to  a  complaint  to  the  Defense  Hotline  concerning  the 
Standard  Procurement  System.  We  considered  management  comments  on  a  draft  of 
this  report  in  preparing  tihe  final  report. 

DoD  Directive  7650.3  requires  that  all  recommendations  and  issues  be  resolved 
promptly.  The  comments  we  received  were  generally  responsive,  but  some 
recommendations  remain  unresolved.  As  a  result  of  management  comments  and  to 
clarify  our  intent,  we  deleted,  added,  and  revised  recommendations.  We  request 
additional  comment,  as  specified  at  the  end  of  each  finding,  by  November  5,  1996. 


Questions  on  the  audit  should  be  directed  to  Ms.  Mary  Ugone,  Audit 
Program  Director,  at  (703)  604-9529  (DSN  664-9529)  (electronic  mail 
address  MUgone@DODIG.OSD.MIL)  or  Mr.  James  Hutchinson,  Audit  Project 
Manager,  at  (703)  604-9530  (DSN  664-9530)  (electronic  mail  address 
JHutchinson@DODIG.OSD.MIL).  See  Appendix  G  for  the  report  distribution. 
The  audit  team  members  are  listed  inside  the  back  cover. 


Robert  J.Xieberman 
Assistant  Inspector  General 
for  Auditing 
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Allegations  to  the  Defense  Hotline  Concerning 
the  Standard  Procurement  System 


Executive  Summary 


Introduction.  The  DoD  is  acquiring  the  Standard  Procurement  System  (SPS),  an 
automated  information  system  that  is  intended  to  implement  a  standard  procurement 
process  and  to  provide  uniform  support  to  all  DoD  procurement  organizations.  This 
acquisition  is  managed  by  the  Defense  Procurement  Corporate  Information 
Management  Systems  Center,  a  Defense  Logistics  Agency  orgai^ation.  The  program 
manager  plans  to  obtain  commercially  available  software  and  to  &st  install  the  software 
at  organizations  that  have  little  or  no  automated  procurement  system  support. 
Functional  enhancements  and  interfaces  with  other  DoD  automated  information  systems 
will  be  accomplished  in  later  software  releases.  When  completed,  the  SPS  will  serve 
about  48,000  users  at  about  1,000  DoD  procurement  organizations.  We  performed  the 
audit  in  response  to  a  complaint  to  the  Defense  Hotline  that  the  planned  strategy  to 
acquire  the  SPS  contained  major  flaws.  The  specific  allegations  and  audit  results  are 
shown  in  Appendix  D,  and  audit  results  are  discussed  in  Part  I  of  this  report. 

Audit  Objectives.  The  primary  audit  objective  was  to  determine  the  validity  of  the 
allegations  made  to  the  Defense  Hotline  concerning  the  acquisition  of  the  SPS.  We 
determined  whether  some  specific  functional  requirements  would  be  met,  evaluated  the 
SPS  acquisition  strategy,  and  determined  whether  system  testing  plans  were  adequate. 
We  also  reviewed  the  adequacy  of  the  management  control  program  as  it  applied  to  the 
planning,  development,  and  execution  of  the  SPS. 

Audit  Results.  The  audit  did  not  substantiate  the  allegations  concerning  specific 
functional  requirements.  Also,  allegations  regarding  the  SPS  acquisition  strategy  and 
testing  plans  had  limited  merit  (Appendix  D).  Although  management  has  limited 
financial  risks,  other  aspects  of  the  SPS  program  involve  substantial  risk. 

o  The  strategy  for  acquiring  the  SPS  adds  considerable  risk  to  the  program 
(Finding  A).  Asa  result,  the  needs  of  SPS  users  may  not  be  met  and  actual  costs  could 
exceed  proposed  costs  because  vendors  do  not  have  well-defined  requirements  and  will 
find  it  difficult  to  provide  realistic  and  comprehensive  cost  proposals. 

o  Inadequate  SPS  testing  strategies  increased  the  risk  that  the  SPS  may  not 
meet  user  requirements  (Finding  B). 

The  recommendations  in  Finding  A  should  result  in  monetaty  benefits,  but  we  could 
not  quantify  the  amounts,  which  are  dependent  on  future  review  results  and  associated 
management  decisions. 

Summary  of  Recommendations.  We  recommend  obtaining  Milestone  Decision 
Authority  agreement  with  the  SPS  program  plan  to  mitigate  risks  related  to  the  contract 
methodology  and  to  determine  whether  the  current  SPS  acquisition  methodology  or  an 
alternative  should  be  used  to  satisfy  later  SPS  increments.  We  further  recommend 
determining  the  most  cost-beneficial  deployment  approach  and  incorporating  Shared 


Data  Warehouse  plans  in  the  SPS  Program  Management  Plan.  We  also  recommend 
developing  and  validating  operational  performance  requirements  and  amending  testing 
schedules  to  include  testing  of  the  Shared  Data  Warehouse  and  to  provide  adequate  time 
for  test  planning  and  review. 

Management  Comments.  The  Assistant  Secretary  of  Defense  (Command,  Control, 
Commumcations,  and  Intelligence)  generally  agreed  that  the  SPS  program  has 
acquisition  risks  and  the  lack  of  operational  requirements  increases  the  risk  of  user 
requirements  not  being  met.  The  Director,  Defense  Procurement,  did  not  specifically 
comment  on  the  findings,  but  disagreed  with  several  topical  discussions.  The  Director, 
Defense  Logistics  Agency,  partially  concurred  with  the  findings,  but  nonconcurred  with 
most  topical  discussions. 

Audit  Response.  As  a  result  of  management  comments,  we  revised  portions  of  the 
finding  regarding  the  strategy  for  acquiring  the  SPS.  We  revised  and  added 
recommendations  related  to  oversight  of  the  SPS  program  because  management  has 
taken  actions  to  limit  financial  risks.  We  also  revised  recommendations  regarding  SPS 
operational  requirements  to  clarify  our  intent.  A  discussion  of  management  comments 
and  audit  responses  regarding  the  recommendations  is  in  Part  I  of  &e  report.  A 
summary  of  management  comments  and  audit  responses  on  the  findings  is  provided  in 
Appendix  E.  The  complete  text  of  management  comments  is  in  Part  III.  We  ask  that 
the  Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 
Intelligence);  the  Director,  Defense  Procurement;  and  the  Director,  Defense  Logistics 
Agency,  provide  written  comments  on  the  final  report  by  November  5,  1996. 
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Part  I  -  Audit  Results 


Audit  Results 


Audit  Background 


Purpose  of  the  Standard  Procurement  System.  DoD  is  acquiring  the 
Standard  Procurement  System  (SPS),  an  automated  information  system  that  is 
intended  to  provide  uniform  support  to  all  DoD  procurement  organizations. 
The  SPS  program  costs^  are  estimated  to  be  about  $330  million;  the  SPS  will 
cost  about  $4.1  billion  through  FY  2010.  The  SPS  program  is  managed  by  the 
Defense  Procurement  Corporate  Information  Management  Systems  Center 
(DPCSC),  a  Defense  Logistics  Agency  organization. 

The  SPS  will  eventually  replace  about  70  legacy^  automated  information 
systems  that  cost  about  $300  million  annually  to  operate  and  maintain.  DoD 
Components  use  procurement  legacy  systems,  including  two  migration^ 
systems,  to  support  contract  award  and  contract  administration  functions.  The 
two  migration  systems  are  the  Defense  Logistics  Agency  Pre-Award  Contracting 
System  and  the  Mechanization  of  Contract  Administration  Services  System. 
The  latter  system  will  be  needed  until  the  SPS  can  interoperate  with  automated 
information  systems  for  accounting  and  finance.  As  presently  funded,  the  SPS 
is  scheduled  to  replace  the  Mechanization  of  Contract  Administration  Services 
System  by  FY  2004. 

SPS  Program  Evolution.  To  implement  the  Corporate  Information 
Management  (CIM)  initiative  throughout  the  DoD  procurement  community,  in 
October  1991,  the  Director,  Defense  Procurement,  formed  the  Procurement 
CIM  Council  (the  Council).  To  provide  user  community  representation,  the 
Council  is  composed  of  senior  procurement  officials  from  the  Military 
Departments  and  Defense  agencies.  The  Council  first  focused  on  defining  an 
improved  DoD  procurement  process.  In  1993,  the  Deputy  Secretary  of 
Defense^  changed  that  focus  to  the  designation  and  development  of  migratory 
automated  information  systems  to  support  the  contract  award  and  contract 
administration  functions  throughout  DoD. 

After  the  migration  systems  were  selected,  the  Council  determined  that 
development  and  funding  projections  would  not  support  the  enhancement  and 
timely  DoD-wide  implementation  of  the  migration  systems.  In  1994,  hoping  to 


^Program  Costs  are  those  expenditures  directly  related  to  SPS  definition, 
design,  development,  and  deployment.  The  costs  include  the  cost  for  the 
Program  Management  Office  and  the  costs  to  acquire,  develop,  and  deploy  each 
increment  of  the  SPS. 

^Existing. 

^An  existing  or  planned  and  approved  automated  information  system  that  has 
been  designated  to  support  a  functional  process  DoD-wide. 

^Deputy  Secretary  of  Defense  Memorandum,  October  13  1993,  Subject: 
Accelerated  Implementation  of  Migration  Systems,  Data  Standards,  and  Process 
Improvement. 
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save  time  and  money  on  internally  developing  a  standard  DoD  procurement 
system,  the  Council  gathered  information  about  commercially  available 
automated  procurement  systems.  As  a  result  of  subsequent  vendor 
demonstrations,  the  Coimcil  determined  that  commercially  available  automated 
information  systems  could  meet  most  contract  award  requirements  of  DoD 
procurement  organizations  and  that  sufficient  commercial  capability  existed  to 
conduct  a  competitive  acquisition. 

In  October  1994,  the  Director,  Defense  Procurement,  accepted  the  Navy  offer 
to  conduct  the  SPS  procurement  and  suggested  specific  nominees  for 
appointment  to  the  source  selection  organization.  Those  nominees  reflect  joint 
end  user  interests  of  the  Office  of  the  Secretary  of  Defense,  Military 
Departments,  and  Defense  agencies.  Additionally,  the  Director,  Defense 
Procurement,  stated  that  the  SPS  acquisition  should  be  conducted  in  accordance 
with  the  Federal  Acquisition  Streamlining  Act  of  1994  (FAS A). 

Established  in  November  1994,  the  DPCSC  evaluated  and  compared  alternatives 
for  SPS  development.  Those  alternatives  were  detailed  in  a  DPCSC  economic 
analysis,  which  showed  that  the  acquisition  of  a  commercially  based  alternative 
would  be  more  cost-effective  than  DoD  development  of  the  SPS.  The  economic 
analysis  did  not  consider  any  particular  contracting  methodology  in  obtaining  a 
commercially  based  alternative. 

Acquisition  Approach.  One  objective  of  FASA  is  the  Government  use  and 
acquisition  of  commercial  products  whenever  possible.  Also,  FASA  redefined 
"commercial  item"  to  include  a  product  that  is  not  yet  available  in  the 
commercial  marke^lace,  but  will  be  available  in  time  to  satisfy  delivery 
requirements.  Additionally,  minor  modifications  may  be  made  to  that  item  to 
meet  Government  requirements.  The  SPS  contracting  officer  determined  that 
commercially  available  procurement  software  qualified  as  a  commercial  item; 
all  SPS  source-selection  principals  agreed  with  that  determination. 

While  commercial  software  products  could  provide  some  basic  procurement 
functionality,  both  SPS  program  and  acquisition  officials  recognized  that  none 
of  those  products  could  meet  all  DoD  procurement  requirements.  Accordingly, 
the  selected  vendor  will  have  to  alter  its  software  to  provide  additional 
functionality,  to  interface  with  other  DoD  automated  information  systems,  and 
to  operate  within  the  DoD  information  infrastructure.  SPS  program  officials 
refer  to  that  altered  software  as  "commercially  derived"  software.  The  SPS 
acquisition  approach  has  no  precedent  in  DoD. 

The  SPS  contracting  officer  issued  a  solicitation  on  October  30,  1995,  and  has 
received  proposals  on  that  solicitation.  Responding  vendors  must  demonstrate 
the  capability  of  their  software  to  meet  functional  requirements.  Based  on 
demonstrated  results  and  other  factors,  the  contracting  officer  will  award  a 
contract  to  "one  or  more"  of  those  vendors  for  further  evaluation,  validation, 
and  acceptance  testing  purposes.  After  the  additional  evaluation,  DoD  will 
select  the  final  SPS  contractor  by  exercising  the  next  contract  option  with  the 
best  overall  contractor.  DoD  has  reserved  the  right  to  not  exercise  contract 
options  with  all  potential  SPS  contractors. 
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Besides  providing  the  necessary  software,  the  selected  vendor  will  also  install 
and  maintain  the  software  and  will  provide  training  to  about  48,000 
procurement  personnel  at  about  1,000  sites  throughout  the  world.  The  DoD 
Components  will  have  to  provide  the  necessary  computers  and  associated 
support  software  and  all  other  necessary  infrastructure  parts,  such  as 
communications . 

Changes  to  SPS  Solicitation.  The  SPS  contracting  officer  amended  the  SPS 
solicitation  to  reflect  its  intent  to  award  an  indefinite  delivery,  indefinite 
quantity  contract  rather  than  the  requirements  type  of  contract  originally 
planned.  Additionally,  specific  requirements  unique  to  the  Defense  Contract 
Management  Command  were  deleted  from  the  solicitation  in  response  to  vendor 
concerns  that  those  unique  requirements  could  not  reasonably  be  accommodated 
with  a  commercial  software  package  without  additional  software  development. 

Oversight  of  the  SPS.  The  SPS  qualifies  as  a  "major"  automated  information 
system  and  is  subject  to  milestone  review  and  approval  by  the  DoD  Major 
Automated  Information  Systems  Review  Council  (MAISRC).  The  MAISRC 
Chair,  the  Assistant  Secretary  of  Defense  (Command,  Control, 
Communications,  and  Intelligence),  is  also  the  SPS  milestone  decision  authority 
and  provides  oversight  of  the  SPS  program.  The  MAISRC  Chair  approved  the 
SPS  for  Milestone  I  on  August  4,  1995,  and  must  approve  a  Milestone  II/III 
before  each  phase  of  SPS  deployment.^  The  first  deployment  is  scheduled  to 
begin  May  1997,  after  selection  of  the  final  SPS  vendor. 

DoD  Guidance.  DoD  acquisition  and  life-cycle  guidance  and  requirements 
applicable  to  the  SPS  program  changed  with  the  issuance  of  the  DoD  Directive 
5000.1,  "Defense  Acquisition,"  March  15,  1996,  and  DoD  Regulation 
5000.2-R,  "Mandatory  Procedures  for  Major  Defense  Acquisition  Programs 
(MDAPS)  and  Major  Automated  Information  System  (MAIS)  Acquisition 
Programs,"  March  15,  1996.  DoD  Directive  5000.1  and  DoD  Regulation 
5000.2  consolidated  and  replaced  previous  DoD  guidance,  which  contained 
different  acquisition  and  life-cycle  management  requirements  and  procedures  for 
mission  critical  computer  resources  and  automated  information  systems.  Until 
March  15,  1996,  the  following  DoD  guidance  applied  to  the  SPS  program: 

o  DoD  Directive  5000.1,  "Defense  Acquisition,"  February  23,  1991; 

0  DoD  Instruction  5000.2,  "Defense  Acquisition  Management  Policies 
and  Procedures,"  February  23,  1991; 

0  DoD  Manual  5000.2-M,  "Defense  Acquisition  Management 
Documentation  and  Reports,"  February  23,  1991; 

0  DoD  Directive  8120.1,  "Life-Cycle  Management  of  Automated 
Information  Systems  (AISs),"  January  14,  1993; 


^Deployment  refers  to  software  installation,  training  of  users,  and  all  steps 
necessary  to  gain  user  acceptance  of  the  SPS. 


4 


Audit  Results 


o  DoD  Instruction  8120.2,  "Automated  Information  System  (AIS)  Life- 
Cycle  Management  (LCM)  Process,  Review,  and  Milestone  Approval 
Procedures,"  January  14,  1993;  and 

o  DoD  Manual  7920.2-M,  "Automated  Information  Systems  Life-Cycle 
Management  Manual,"  March  1990. 


Audit  Objectives 


The  primary  objective  of  this  audit  was  to  determine  the  validity  of  the 
allegations  in  a  complaint  to  the  Defense  Hotline  concerning  die  SPS. 
Specifically,  we: 

0  determined  whether  the  SPS  would  meet  some  specific  functional 
requirements, 

o  evaluated  the  SPS  acquisition  strategy,  and 

o  determined  whether  system  testing  plans  were  adequate. 

See  Appendix  A  for  a  discussion  of  the  audit  scope  and  methodology  and  for  the 
results  of  the  review  of  the  management  control  program.  Appendix  B  contains 
a  summary  of  prior  coverage  related  to  the  audit  objectives.  Appendix  C 
provides  additional  comments  about  the  SPS  acquisition  program  baseline  and 
SPS  advisory  councils.  Appendix  D  discusses  the  specific  allegations  and  our 
audit  results  pertaining  to  each  allegation. 
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Finding  A.  Strategy  for  Acquiring  the 
Standard  Procurement  System 

The  strategy  for  acquiring  the  SPS  adds  considerable  risk  to  the  SPS 
program,  estimated  at  a  life-cycle  cost  of  $4.1  billion  through  the 
year  2010,  The  fixed-price  contracting  methodology  used  for 
commercial  items  is  risky  because  SPS  functional  requirements  in  the 
solicitation  are  too  broad  and  because  existing  commercial  software 
requires  substantial  software  development  to  achieve  full  SPS  functional 
capability.  Also,  the  SPS  solicitation  does  not  sufficiently  define  site 
requirements.  Further,  the  program  manager  did  not  quantitatively 
analyze  alternative  deployment  approaches  or  stress  the  significance  of 
the  Shared  Data  Warehouse  in  program  plans  to  assure  the  ultimate 
success  of  the  SPS  program.  As  a  result,  the  needs  of  SPS  users  may 
not  be  met  and  actual  costs  could  exceed  proposed  costs  because  vendors 
will  have  difficulty  in  providing  realistic  and  comprehensive  cost 
proposals  without  well-defined  functional  and  site  requirements. 


Acquisition  of  Conmiercial  Items 


Federal  Acquisition  Streamlining  Act  of  1994  (FAS A).  The  FASA 
establishes  a  streamlined  approach  to  acquisition  that  more  closely  resembles  the 
commercial  markeq)lace  and  that  encourages  the  acquisition  of  commercial 
items.  Federal  Acquisition  Regulation  part  12,  "Acquisition  of  Commercial 
Items, "  implements  FASA  and  prescribes  the  policies  and  procedures  unique  to 
the  acquisition  of  commercial  items.  Part  12  requires  that: 

o  fixed-price  contracts  be  used  to  acquire  commercial  items  and 

0  requirements  be  defined  to  optimize  conunercially  available 
technology  rather  than  to  satisfy  detailed  technical  specifications. 

SPS  Acquired  as  a  Commercial  Item.  Contracting  officials  determined  that 
the  SPS  could  be  acquired  as  a  commercial  item  and  that  Federal  Acquisition 
Regulation,  part  12,  procedures  applied  in  relation  to  using  fixed-price  contracts 
to  acquire  commercial  items.  Acquiring  the  SPS  as  a  commercial  item  using  a 
fixed-price  contract  is  risky  because  no  known  commercial  software  product 
meets  all  SPS  requirements  and  operates  in  an  environment  as  diverse  as  that  of 
DoD. 

The  use  of  part  12  procedures  to  acquire  a  commercial  item  should  involve  little 
risk  to  the  Government.  For  example,  the  purchase  of  computer  hardware 
involves  little  risk  when  it  is  readily  available  in  the  commercial  marketplace 
and  when  the  performance  specifications  and  capabilities  are  well  established. 
The  SPS  requirements  in  the  solicitation  were  deliberately  broad  to  encourage 
competition,  to  satisfy  part  12  requirements,  and  to  provide  flexibility  for 
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vendor-unique  solutions.  However,  generalized  requirements  increase  the 
likelihood  diat  the  SPS  will  not  meet  specific  user  needs  without  substantial 
follow-on  software  development  as  SPS  is  progressively  deployed  to  user  sites. 


Need  for  Software  Development 


SPS  Software  Development.  No  known  commercial  software  product  can 
meet  total  SPS  requirements  at  contract  award.  Program  management  officials 
believe  that  vendor  proposals  will  satisfy  from  60  to  75  percent  of  SPS 
functional  requirements.  The  program  manager  anticipates  that  potential 
bidders  will  develop  the  remaining  functionality  and  that  the  negotiated 
licensing  fees  will  include  associated  costs.  The  FAS  A  permits  minor  changes 
to  a  commercial  item  so  long  as  the  commercial  item  will  be  available  when 
needed. 

Another  indicator  that  substantial  software  development  is  anticipated  is  the 
requirement  in  the  solicitation  for  potential  bidders  to  submit  a  software  process 
questionnaire  to  the  contracting  officer.  Procurement  officials  will  evaluate  and 
establish  the  maturity  of  a  bidder's  software  development  and  maintenance 
process.  This  requirement  is  not  typical  for  commercial  software  acquisitions. 
In  the  commercial  markeq)lace,  &e  quality  of  software  products  is  largely 
sustained  through  public  acceptance.  The  requirement  is  a  strong  indicator  that 
the  Government  anticipates  substantial  software  development  for  the  SPS  to 
satisfy  user  requirements. 

Software  Engineering  Support.  The  solicitation  shows  an  estimate  of 
330,000  hours  needed  for  software  engineering  support  over  the  life  of  the  SPS. 
Using  a  composite  rate  of  $100  per  hour,  we  estimated  the  cost  of  that  support 
to  be  $33  million.  The  term  software  engineering  often  refers  to  a  disciplined 
methodology  for  software  design  and  development.  Although  the  solicitation 
contains  no  contract  line  item  for  software  development,  the  solicitation  contains 
a  line  item  for  software  engineering  support.  Software  development  needed  for 
the  SPS  to  comply  with  changes  in  regulations  or  laws  is  included  in  software 
engineering  support.  An  example  of  needed  software  development  is  meeting 
the  requirement  to  convert  past  performance  data  from  numerous  DoD  systems, 
both  automated  and  manual,  into  the  SPS.  This  conversion  effort  will  be 
charged  to  software  engineering  support  and  may  require  more  than  minor 
software  development  to  achieve.  Vendors  will  find  it  difficult  to  estimate  the 
associated  costs  because  the  level  of  effort  and  extent  of  development  are 
unknown. 

Functional  Requirements.  SPS  program  officials  anticipated  that  the  SPS 
software  proposed  would  not  meet  all  requirements  initially,  but  that  the 
proposed  software  would  need  to  be  modified  to  meet  the  diverse  requirements 
of  &e  DoD  procurement  community.  We  believe  the  software  will  also  require 
modification  due  to  the  vagueness  of  fiinctional  requirements  as  stated  in  the 
solicitation.  The  procurement  community  initially  submitted  about  700  detailed 
user  requirements.  To  implement  FASA  provisions,  the  Source  Selection 
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Advisory  Council  used  part  12  procedures  and  consolidated  those  700 
requirements  into  about  300  broadly  stated  functional  requirements  in  the  SPS 
solicitation.  In  meeting  the  intent  of  FASA,  the  functional  requirements  in  the 
solicitation  contained  few  detailed  specifications.  However,  the  stated 
requirements  may  no  longer  accurately  reflect  user  requirements.  The  evolution 
of  the  requirement  for  the  SPS  to  provide  past  performance  data  illustrates  how 
functional  requirements  may  have  become  so  vague  that  those  requirements  no 
longer  reflect  the  full  needs  of  the  user. 

Past  Performance.  The  requirement  for  SPS  to  provide  past 
performance  data  is  one  of  the  primary  needs  in  the  SPS  Mission  Need 
Statement,  approved  May  1995.  The  SPS  solicitation  contains  at  least  two 
requirements  relating  to  past  performance:  "perform  contractor  assessment"  and 
"evaluate  offers."  The  first  draft  solicitation,  dated  April  14,  1995,  stated: 

The  system  [SPS]  shall:  (A21-D)  Aggregate  contract  performance 
information  into  contractor  performance  summary  reports.  Use  these 
summary  reports,  along  with  other  Contractor  information ...  to 
construct  and  populate  a  vendor  rating  system.  Allow  for  review  and 
editing  by  the  authorized  user  prior  to  posting  for  general  access. 

(A21-E)  Permit  authorized  user  to  construct  performance  summary 
reports,  as  required,  based  on  specific  parameters.  These  reports  will 
be  stored  as  read-only  access  and  indexed  to  multiple,  applicable 
subject  areas. 

However,  as  subsequent  versions  of  the  solicitation  were  released,  the 
requirement  became  less  specific.  The  same  requirement  in  the  final 
solicitation,  dated  October  30,  1995,  states:  "The  system  [SPS] 

shall:  (B)  aggregate  contract  performance  information  into  contractor 
performance  summary  reports,  and  use  these  summary  reports  along  with  other 
contractor  information  to  create  vendor  rating  summary  reports. " 

The  final  solicitation  deleted  requirements  to  construct  and  populate  a  vendor 
rating  system,  to  allow  for  user  review  and  edit,  to  provide  the  capability  to 
permit  user-constructed  performance  summary  reports,  and  to  store  those 
summary  reports  as  read-only  access  that  can  be  indexed  to  multiple,  applicable 
subject  areas. 

Meeting  Functional  Requirements.  Although  FASA  anticipates  and 
allows  for  minor  modifications  to  a  commercial  item,  we  believe  the  effort 
needed  to  achieve  full  SPS  functionality  is  significant  and,  therefore,  a  risk 
factor.  More  importantly,  the  SPS  requirements  are  too  broad  for  potential  SPS 
contractors  to  precisely  estimate  the  costs  involved  in  meeting  stated 
procurement  needs.  As  a  result,  the  Government  assumes  greater  program  risk 
that  those  requirements  will  lead  to  additional  software  development.  Further, 
we  believe  that  because  the  procurement  community  and  its  user  needs  are  so 
diverse,  it  may  not  be  possible  or  desirable  to  meet  those  needs  with  a 
commercial  product  procured  within  the  constraints  of  a  fixed-price  contract. 
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SPS  Site  Requirements 


About  1,000  sites  expect  to  use  the  SPS  and  each  site  has  distinct  requirements 
within  which  the  SPS  must  operate.  However,  the  SPS  solicitation  does  not 
contain  detailed  requirements  for  each  site  that  will  receive  the  SPS.  Therefore, 
vendors  will  have  to  develop  their  cost  proposals  based  on  their  best  estimates, 
because  information  about  the  sites  is  incomplete  or  unknown. 

Needed  SPS  Interfaces.  The  solicitation  requires  that  the  SPS  interface  with 
numerous  existing  automated  information  systems  as  well  as  with  systems  not 
yet  developed.  The  existing  systems  include  other  functional  legacy  systems  of 
the  DoD  Components,  procurement  systems  and  statistical  analysis  software 
used  at  each  procurement  site,  and  the  other  DoD  standard  functional  systems 
that  have  been  developed.  Most  existing  systems  with  which  the  SPS  will  have 
to  interface  have  been  identified,  but  those  systems  lack  complete  system 
documentation.  Therefore,  SPS  bidders  will  be  unable  to  rely  on  existing 
system  documentation  to  identify  all  necessary  SPS  interfaces.  The  SPS  will 
also  have  to  provide  interfaces  with  other  evolving  or  future  DoD  and 
Government-wide  systems.  Accordingly,  the  effort  to  identify  the  necessary 
information  for  SPS  interfacing  purposes  is  not  well-defined  and  may  be 
extensive.  The  lack  of  well-defined  interface  information  will  further  affect  the 
vendors'  ability  to  develop  realistic  estimates  of  the  effort  required. 

Diversity  of  User  Requirements.  Each  site  varies  by  size,  mission,  work  load, 
and  assignments.  To  help  meet  those  diverse  requirements,  the  functionality  of 
the  SPS  must  be  extremely  flexible.  For  example,  the  SPS  will  provide  standard 
templates  to  capture  and  store  data  values  or  text  fields.  Users  at  each  site  must 
be  able  to  modify  those  templates  to  meet  specific  needs.  The  SPS  will  also 
have  to  adapt  to  varying  work  flows  and  automatic  assignments,  which  will  also 
vary  by  site.  Although  the  need  for  SPS  flexibility  is  toown,  tihe  extent  of  that 
flexibility  is  not.  Similarly,  perspective  bidders  will  be  unable  to  determine 
whether  inherent  limitations  in  their  software  can  accommodate  specific  needs  at 
each  SPS  site. 

Further,  the  SPS  solicitation  and  initial  analyses  indicate  that  substantial 
programming  will  be  required.  Additionally,  SPS  requirements  are  very 
broadly  stated  and  do  not  contain  well-defmed  SPS  site  requirements. 
Consequently: 

o  sound  and  comprehensive  cost  proposals  will  be  difficult  to  prepare, 

0  SPS  user  needs  may  not  be  met,  and 

o  proposed  costs  may  be  exceeded. 
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SPS  Acquisition  Strategy 


DoD  Regulation  5000.2-R,^  "Mandatory  Procedures  for  Major  Defense 
Acquisition  Programs  (MDAPS)  and  Major  Automated  Information  System 
(MAIS)  Acquisition  Programs,"  March  15,  1996,  requires  that  program 
managers  shall 


.  .  .  develop  and  document  an  acquisition  strategy  that  shall  serve  as 
the  roadmap  for  program  execution  from  program  initiation  through 
post  production  support.  A  primary  goal  in  developing  an  acquisition 
strategy  shall  be  to  minimize  the  time  and  costs  of  satisfying  an 
identified,  validated  need,  consistent  with  common  sense  and  sound 
business  practices.  The  acquisition  strategy  shall  evolve  through  an 
iterative  process  and  become  increasingly  more  definitive  in 
describing  the  relationship  of  the  essential  elements  of  a  program. 

Essential  elements,  in  this  context  include,  but  are  not  limited  to, 
sources,  risk  management,  cost  as  an  independent  variable,  contract 
approach,  management  approach,  environmental  considerations,  and 
source  of  support. 

DoD  Regulation  5000. 2-R  also  directs  program  managers  to  use  the  acquisition 
strategy  to  meet  the  needs  of  the  user  community.  Specifically, 

The  acquisition  strategy  shall  be  tailored  to  meet  the  specific  needs  of 
individual  programs,  including  consideration  of  incremental  (block) 
development  and  fielding  strategies.  The  benefits  and  risks  associated 
with  reducing  lead  time  through  concurrency  shall  be  specifically 
addressed  in  tailoring  the  acquisition  strategy. 

Incremental  Program  Strategy.  The  SPS  program  manager  defined  an 
incremental  program  strategy  for  the  SPS  yet  did  not  define  discrete  increments 
for  delivery,  implementation,  and  testing.  The  DoD  Instruction  8120.2,^ 
"Automated  Information  System  (AIS)  Life-Cycle  Management  (LCM)  Process, 
Review,  and  Milestone  Approval  Procedures,"  January  14,  1993,  characterized 
an  incremental  program  strategy  as  "the  acquisition,  development,  and 
deployment  of  fonctionality  through  a  number  of  clearly  defined  system 
increments  that  stand  on  their  own."  The  SPS  program  manager  used  the 
existing  procurement  systems  that  the  SPS  would  replace  to  identify  specific 
increments  rather  than  identifying  discrete  functional  increments.  That 
approach  considered  the  SPS  priority  to  add  automation  to  organizations  with 
little  or  no  current  automation,  thereby  reducing  SPS  technicaJ  risk.  Because 
those  organizations  have  little  or  no  automation  and  do  not  require  interfaces  to 


^DoD  Directive  5000.1,  DoD  Directive  8120.1,  and  DoD  Instruction  5000.2 
(which  are  all  discussed  later  in  the  report)  have  been  canceled  and  replaced 
with  a  revised  DoD  Directive  5000.1  and  DoD  Regulation  5000.2-R  as  of 
March  15,  1996.  The  SPS  program  documentation  reviewed  in  this  audit  were 
covered  by  the  earlier  regulations. 

^DoD  Instruction  8120.2  was  canceled  and  replaced  with  a  revised  DoD 
Directive  5000.1  and  DoD  Regulation  5000.2-R  as  of  March  15,  1996. 
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other  automated  systems,  that  approach  involves  minimal  technical  risk. 
Although  that  approach  should  mitigate  some  technical  risk,  it  does  not  assure 
that  incremental  functionality  will  be  provided  to  the  DoD  procurement 
community. 

Analysis  of  Alternative  Deployment  Approaches.  The  SPS  program  manager 
did  not  quantitatively  analyze  the  costs  and  benefits  of  other  deployment 
approaches.  Instead,  the  only  factors  the  program  manager  seriously  considered 
were  technical  complexity  and  expected  early  benefits.  Using  those  criteria,  the 
program  manager  expected  to  install  the  SPS  at  organizations  using  manual  or 
semi-automated  procedures  first.  Those  organizations  do  not  require  interfaces 
with  other  DoD  systems  and,  therefore,  pose  less  technical  risk.  Because  those 
organizations  have  no  or  little  automated  support,  they  will  benefit  from  the 
automation  of  whatever  SPS  functional  capabilities  exist  at  that  time.  The 
program  manager  quantified  the  expected  benefits  of  the  preferred  deployment 
approach,  but  he  did  not  closely  examine  the  costs  and  benefits  of  alternative 
SPS  deployment  approaches.  One  alternative  approach  would  be  to  focus  on 
first  replacing  procurement  legacy  systems. 

The  program  manager  estimated  that  the  procurement  legacy  systems  cost  about 
$300  million  a  year  to  operate.  The  SPS  is  projected  to  replace  all  those 
systems  by  FY  2004.  If  those  systems  are  replaced  by  FY  2003,  DoD  could 
avoid  legacy  system  expenditures  of  $300  million  during  FY  2004.  We 
recognize  that  DoD  may  still  incur  costs  associated  with  replacement  of  those 
legacy  systems,  but  those  costs  have  not  been  identified  by  the  program 
manager.  Because  the  SPS  program  manager  has  neither  performed  any 
quantitative  analyses  of  the  costs  and  benefits  related  to  other  deployment 
approaches  nor  determined  the  most  effective  development  approach,  the  SPS 
may  not  represent  the  best  value  to  DoD. 

Risks  Associated  with  SPS  Deployment.  The  risks  associated  with  the  base 
year  and  option  year  1  of  the  contract  for  the  SPS  are  minimal.  However,  the 
risks  associated  with  the  SPS  deployment  in  option  years  2  through  4  are  high. 
Table  1  shows  the  number  of  sites  and  users  that  will  receive  SPS  for  each 
option  period. 


Table  1.  SPS  Deployment  Schedule 


Contract  Year 

Sites 

Users 

Base  Year 

15 

1,509 

Option  One 

408 

8,854 

Option  Two 

357 

14,369 

Option  Three 

108 

8,136 

Option  Four 

102 

14,945 

Totals 

990 

47,813 

Planned  SPS  Deployment.  The  preferred  method  for  deploying  the  SPS  is 
incremental,  beginning  with  the  base  contract  award.  The  risks  associated  wi& 
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the  base  award  are  minimal.  In  the  base  year,  15  candidate  sites  (with  manual 
and  automated  systems)  will  act  as  the  SPS  test  bed  for  demonstration  and 
validation  testing  of  two  or  more  competitive  vendors.  The  Source  Selection 
Advisory  Council  will  use  those  test  results  to  recommend  the  vendor  best 
qualified  to  meet  SPS  requirements.  Any  options  exercised  would  be  awarded 
to  the  winning  vendor. 

First  Option  Year.  The  Comptroller  of  the  Department  of  Defense  and 
program  management  officials  believe  that  the  SPS  will  help  solve  DoD 
problems  with  unmatched  disbursements  and  negative  unliquidated  obligations. 
Because  the  SPS  will  automate  manual  systems,  the  SPS  will  also  eliminate  the 
need  to  reenter  contract  data,  thereby  eliminating  most  "keystroke"  errors  that 
have  caused  unmatched  disbursements  and  negative  unliquidated  obligations 
throughout  the  DoD.  In  the  first  option  year,  the  SPS  will  be  installed  at  more 
than  400  manual  and  semiautomated  organizations,  primarily  Navy 
organizations.  Of  the  Military  Departments,  the  Navy  has  the  highest 
percentage  of  unmatched  disbursements,  which  is  attributed  to  its  primarily 
manual  systems.  In  the  first  option  year,  the  SPS  will  provide  those 
organizations  with  the  capability  to  electronically  send  contract  data  to  the 
Defense  Finance  and  Accounting  Service.  To  date,  the  Defense  Finance  and 
Accounting  Service  obtains  that  data  in  hardcopy  format  and  must  manually  key 
the  data  into  its  own  systems.  The  risks  associated  with  this  deployment  for  the 
SPS  are  minimal. 

Option  Years  2  through  4.  Beginning  with  the  second  option  year,  we 
believe  the  functional  and  site  requirement  factors  previously  discussed  will 
present  high  risks  to  the  SPS  program.  The  SPS  will  be  required  to  replace 
18  known  automated  information  systems  during  option  years  2  through  4. 
Additionally,  differing  site  architectures,  the  need  for  multiple  interfaces,  and 
the  diversity  of  user  requirements  increase  the  likelihood  that  software 
development  will  be  needed.  Further,  the  submitted  proposals  indicated  that 
none  of  the  bidders  have  the  total  desired  functional  capability  in  their 
commercial  products.  The  SPS  will  not  be  installed  at  a  site  unless  the  SPS  can 
provide  that  organization  with  functional  capability  at  least  equal  to  what 
already  exists  there. 

SPS  Program  Options.  Although  the  risks  in  option  years  2  through  4  are 
high,  the  DoD  is  not  obligated  to  incur  that  risk.  The  DoD  has  the  following 
primary  options: 

0  cancel  the  existing  solicitation  and  recompete  using  a  different 
contracting  methodology  or 

o  continue  with  the  existing  contract  type  through  option  year  1  and 
resolicit  with  a  different  contracting  methodology  to  obtain  option  years  2 
through  4  or 

0  continue  with  the  SPS  development  as  planned,  fiilly  knowledgeable 
of  the  risks  involved. 
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The  SPS  solicitation  does  not  obligate  the  Government  to  acquire  the  SPS  for 
any  minimum  number  of  sites. 


SPS  Shared  Data  Warehouse 


Importance  of  Shared  Data.  The  SPS  Program  Management  Plan  did  not 
adequately  reflect  the  importance  of  the  SPS  Shared  Data  Warehouse  to  the 
ultimate  success  of  the  SPS  program.  The  SPS  mission  need  statement,  dated 
May  1995,  requires  the  SPS  to  "facilitate  the  DoD-wide  integration  of  [standard 
data]  through  the  implementation  of  standard  processes,  standard  shared  data, 
and  electronic  contracting. "  Further,  the  SPS  is  to  "provide  for  improved  data 
management  and  data  integrity  by  electronic  input  of  selected  data  to  a  logically 
shared  data  repository."  To  meet  those  basic  needs,  the  SPS  program  manager 
plans  to  develop  the  SPS  Shared  Data  Warehouse.  The  Shared  Data  Warehouse 
will  essentially  be  the  single  DoD  data  base  for  storing  standardized 
procurement  data  and  will  be  designed  and  developed  by  the  Defense  Logistics 
Agency.  The  Shared  Data  Warehouse  will  be  used  on  a  cross-functional  basis 
throughout  DoD.  For  example,  contract  payment  data  in  the  SPS  Shared  Data 
Warehouse  would  be  used  by  procurement  activities  and  the  Defense  Finance 
and  Accounting  Service,  which  would  use  the  data  to  make  contract  payment. 
Although  vital  to  the  overall  success  of  the  SPS  program,  the  SPS  Program 
Management  Plan  contains  little  discussion  of  plans  to  define  and  establish  the 
SPS  Shared  Data  Warehouse. 

Shared  Data  Warehouse  Plans.  Although  SPS  program  management  officials 
stated  that  the  timely  implementation  of  the  SPS  Shared  Data  Warehouse  was  of 
major  concern,  the  SPS  Program  Management  Plan  neither  includes  an 
implementation  schedule  for  the  SPS  Shared  Data  Warehouse  nor  discusses 
factors  that  may  impede  its  development.  Initial  design  of  the  SPS  Shared  Data 
Warehouse  has  been  completed,  but  substantial  effort  remains.  Functional  user 
requirements  have  not  been  established.  Until  those  requirements  are  defined, 
development  of  the  Shared  Data  Warehouse  cannot  begin.  Additionally, 
because  the  data  are  to  be  shared  among  DoD  functional  areas,  those  functional 
areas  must  agree  on  data  usage.  Reaching  agreement  has  been  difficult, 
however,  because  the  functional  areas  are  at  various  stages  in  developing 
"standard"  automated  information  systems.  We  believe  integrating  the  Shared 
Data  Warehouse  plans  into  the  SPS  Program  Management  Plan  would  provide  a 
clearer  picture  of  the  progress  of  the  SPS  and  would  help  assure  that  all  vital 
SPS  components  are  considered.  That  integration  would  help  lessen  the  risk 
that  the  SPS  program  may  not  meet  the  primary  objectives  of  the  mission  need 
statement. 
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Conclusion 


The  acquisition  of  the  SPS  as  a  commercial  item,  which  requires  fixed-price 
contracting  and  broadly  stated  requirements,  is  risky.  The  SPS  solicitation 
requirements  are  so  broad  that  potential  bidders  may  not  be  able  to  formulate 
accurate  cost  proposals.  It  is  uncertain  whether  the  diverse  needs  of  all  SPS 
users  will  effectively  be  met  or  whether  contract  costs  will  remain  fixed.  The 
SPS  program  manager  needs  to  perform  quantitative  analyses  of  costs  and 
benefits  associated  with  different  deployment  approaches  to  help  support  the 
deployment  strategy  to  be  selected.  The  DoD  should  reexamine  tiie  SPS 
acquisition  and  program  strategies  to  determine  whether  the  inherent  risks  are 
still  acceptable  or,  if  deemed  appropriate,  use  another  contracting  approach  to 
reduce  risks  to  the  Government.  In  addition,  the  program  manager  needs  to 
incorporate  the  SPS  Shared  Data  Warehouse  into  the  SPS  Program  Management 
Plan  and  schedule. 


Management  Comments  on  the  Finding  and  Audit  Response 


Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 
Intelligence).  The  Assistant  Secretary,  as  Ae  Chair  of  the  Major  Automated 
Information  Systems  Review  Council,  generally  concurred  that  the  SPS  program 
has  acquisition  risks.  However,  the  Assistant  Secretary  stated  that,  through  the 
normal  MAISRC  oversight  process,  SPS  technical  risks  had  been  identified  and 
that  MAISRC  staff  assisted  and  supported  the  SPS  Program  Manager  in 
developing  risk  mitigation  strategies. 

Director,  Defense  Procurement.  The  Director,  Defense  Procurement,  did  not 
specifically  concur  or  nonconcur  with  the  finding,  but  did  comment  on  several 
topics  discussed  in  the  finding.  See  Appendix  E  for  a  summary  of  comments 
provided  and  the  audit  response. 

Director,  Defense  Logistics  Agency.  The  Director,  Defense  Logistics  Agency, 
partially  concurred  with  the  finding,  but  nonconcurred  with  most  topics 
discussed  and  provided  extensive  comments  on  the  finding.  A  summary  of 
Defense  Logistics  Agency  comments  and  the  audit  response  are  provided  in 
Appendix  E. 


Recommendations,  Management  Comments,  and  Audit 
Response 


Deleted,  Added,  and  Revised  Recommendations.  As  a  result  of  management 
actions  to  limit  financial  risks,  we  deleted  draft  Recommendation  A.l.a.  The 
final  Recommendation  A.l.a.  was  added  to  help  ensure  that  an  appropriate  risk 
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mitigation  plan  is  developed  for  later  SPS  increments.  We  also  revised 
Recommendations  A.l.b,  A.2.a.,  and  A.2.b.  to  clarify  our  intent  or  recognize 
extenuating  circumstances.  The  complete  text  of  management  comments  is  in 
Part  III. 

A.l.  We  recommend  that  the  Chair,  Major  Automated  Information 
Systems  Review  Council,  for  the  Standard  Procurement  System  program: 

a.  Require  the  Program  Manager  to  obtain  Milestone  Decision 
Authority  agreement  with  the  program  plan  to  mitigate  risks  related  to  the 
contract  methodology  before  the  l^ogram  Manager  authorizes  any  work  to 
meet  requirements  of  contract  option  years  2  through  4. 

b.  At  each  Milestone  approval  review  preceding  contract  option 
years  2  through  4,  reevaluate  the  contracting  methodology  as  a  speciHc 
item  of  interest  and  determine  whether  that  methodology  or  an  alternative 
will  be  used  to  satisfy  contract  option  years  2  through  4. 

Management  Comments.  The  Assistant  Secretary  of  Defense  (Command, 
Control,  Communications,  and  Intelligence)  stated  that  the  SPS  Program 
Manager  has  and  continues  to  establish  appropriate  processes  for  minimizing 
program  risks.  However,  because  of  stated  concerns,  a  MAISRC 
Working-level  Integrated  Product  Team  will  again  reexamine  the  contracting 
approach  within  90  days  of  contract  award. 

Audit  Response.  The  SPS  solicitation  was  amended  to  reduce  risks  associated 
with  the  SPS  contracting  approach.  Because  DoD  financial  risk  has  been 
limited,  we  deleted  the  draft  report  recommendation  that  the  MAISRC  evaluate 
risks  associated  with  the  contracting  methodology  before  contract  award. 
However,  we  believe  that  the  contracting  methodology  continues  to  present 
substantial  risk  to  the  SPS  program  for  contract  option  years  2  through  4. 
Accordingly,  we  revised  Recommendation  A.l.b.  to  help  ensure  that  those  risks 
are  recognized  and  appropriately  evaluated.  We  also  added  Recommendation 
A.l. a.  to  help  ensure  that  SPS  program  risk  mitigation  plans  are  appropriate. 
We  ask  that  the  Assistant  Secretary  provide  comments  on  the  revised  and  added 
recommendations  in  response  to  the  final  report. 

A.2.  We  recommend  that  the  Director,  Defense  Procurement  Corporate 
Information  Management  Systems  Center,  Defense  Logistics  Agency: 

a.  In  coordination  with  the  Director,  Program  Analysis  and 
Evaluation,  quantitatively  determine  the  most  cost-beneHcial  deployment 
approach  and,  if  applicable,  justify  deviations.  If  related  costs  and  benefits 
cannot  be  soundly  quantiHed,  document  the  mitigating  circumstances  and 
factors  considered  in  determining  the  best  overall  SPS  deployment 
approach. 

b.  Include  speciHc  plans  in  the  Standard  Procurement  System 
Program  Management  Plan  to  establish  the  Standard  Procurement  System 
Shared  Data  Warehouse.  That  description  should  include  both  known  and 
anticipated  risk  factors  and  associated  abatement  plans. 
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Management  Comments.  The  Defense  Logistics  Agency  concurred  with  both 
recommendations.  Regarding  the  deployment  approach,  management  stated 
that,  in  accordance  with  existing  DoD  requirements,  the  most  cost-beneficial 
deployment  approach  would  be  determined  and  provided  in  the  SPS  Economic 
Analysis,  which  is  required  for  die  SPS  Milestone  II/IIIA  review  scheduled  for 
March  1997.  Management  will  also  update  the  SPS  Economic  Analysis  to 
document  subsequent  deployment  recommendations  during  the  respective  SPS 
Milestone  II/III  review  required  for  each  deployment  phase.  Regarding  the 
Shared  Data  Warehouse,  management  stated  that  the  SPS  Program  Management 
Plan  would  be  amended  before  March  1997  to  include  the  development  of  the 
Shared  Data  Warehouse. 

Audit  Response.  Although  the  Agency  concurred  with  the  recommendation  to 
determine  the  most  cost-beneficial  deployment  approach,  the  associated  plan  of 
action  is  not  fully  responsive.  Because  the  intended  deployment  is  clearly  set  in 
the  SPS  solicitation,  most  vendor  proposals  are  premised  on  that  approach.  As 
such,  contract  award  and  the  subsequent  "down  select,"  both  of  which  will 
occur  before  the  scheduled  Milestone  II/IIIA  review,  are  also  likely  to 
incorporate  the  intended  deployment  approach.  By  the  time  the  Milestone  II/III 
occurs,  we  believe  it  will  be  too  late  to  seriously  consider  alternative 
deployment  approaches.  Until  the  associated  costs  and  benefits  of  alternate 
approaches  are  developed  and  compared,  potential  financial  savings  remain 
undefined.  We  acknowledge  that  it  is  difficult,  at  best,  to  (piantitatively 
determine  some  cost  and  benefit  factors.  In  recognition  of  that  difficulty,  we 
have  revised  Recommendation  A.2.a.  We  ask  that  the  Defense  Logistics 
Agency  provide  comments  on  the  revised  recommendation  in  response  to  the 
final  report. 

Agency  comments  regarding  Recommendation  A.2.b.  are  fully  responsive  and 
no  further  conunent  is  required. 
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The  SPS  program  manager  has  not  developed  adequate  developmental 
and  operational  test  strategies  for  the  SP^  The  testing  strategy  is 
inadequate  because: 

o  the  Procurement  Corporate  Information  Management  Council 
has  neither  provided  the  SPS  program  management  office  and  the  test 
community  with  user-validated  operational  requirements  nor  defined  the 
content  of  each  functional  increment, 

0  the  compressed  integrated  program  schedule  does  not  allow 
sufficient  time  to  ensure  the  development  of  comprehensive  test  plans, 
and 


o  the  Council  has  not  identified  requirements  to  the  program 
manager  and  test  community  for  the  Shared  Data  Warehouse  portion  of 
the  SPS  program  to  enable  Ae  development  of  a  testing  strategy. 

As  a  result,  the  SPS  program  may  not  meet  user  requirements. 


Mandatory  Testing 


DoD  Directive  8120.1,  "Life-Cycle  Management  of  Automated  Information 
Systems  (AISs),"  January  14,  1993,  directs  that  developmental  and  operational 
tests  for  automated  information  systems  be  conducted  according  to  the  guidance 
in  DoD  Instruction  5000.2,  "Defense  Acquisition  Management  Policies  and 
Procedures,"  February  23,  1991.  DoD  Instruction  5000.2  prescribes  the  policy 
and  procedures  for  testing.  The  format  and  content  for  the  Test  and  Evaluation 
Master  Plan  (TEMP)*  is  in  Draft  DoD  Manual  8120.2-M,  "Automated 
Information  System  Life-Cycle  Management  Manual,"  March  1995. 

The  overall  purpose  of  testing  described  in  DoD  Instruction  5000.2  is  to  provide 
decision  makers  with  essential  information  for  assessment  of  acquisition  risk, 
verify  attainment  of  technical  performance  objectives,  verify  that  systems  are 
operationally  suitable  and  effective,  and  provide  information  to  support 
decisionmaking.  Specifically,  the  purpose  of  developmental  testing  is  to 
identify  potential  operational  and  technological  limitations  of  the  alternative 
concepts  and  design  options  being  pursued,  to  support  the  identification  of  cost- 


^The  Test  and  Evaluation  Master  Plan  documents  the  overall  structure  and 
objectives  of  the  test  and  evaluation  program.  The  TEMP  provides  a 
framework  within  which  to  generate  detailed  test  and  evaluation  plans  and 
document  schedule  and  resource  implications  associated  with  the  test  and 
evaluation  program. 
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performance  trade-offs,  to  support  the  identification  and  description  of  design 
risks,  to  substantiate  that  contract  technical  performance  and  manufacturing 
process  requirement  have  been  achieved,  and  to  support  the  decision  to  certify 
the  system  as  ready  for  operational  test  and  evaluation.  DoD  Instruction  5000.2 
states  that  operational  testing  should  be  structured  to  determine  the  operational 
effectiveness  and  suitability  of  a  system  under  realistic  conditions  and  to 
determine  whether  the  minimum  acceptable  operational  performance 
requirements,  as  specified  in  the  Operational  Requirements  Document,  have 
been  satisfied. 


Operational  Requirements 


The  Critical  Operational  Issues,  Minimum  Acceptable  Operational  Performance 
Requirements,  and  Critical  Technical  Parameters  in  the  December  1,  1995,  SPS 
TEMP  were  not  based  on  user-developed  and  validated  operational 
requirements^  as  directed  by  DoD  Instruction  5000.2.  The  lack  of  SPS 
user-based  operational  requirements  impairs  the  ability  of  the  developmental  and 
operational  testers  to  plan  and  execute  a  comprehensive  test  and  may  result  in 
inadequate  testing  of  technical  and  operational  capabilities. 

User-validated  Requirements.  The  sources  cited  in  the  December  1,  1995, 
TEMP  for  the  measures  of  technical  and  operational  performance  to  be  used  in 
testing  are  the  Test  and  Evaluation  Working  Group,  the  Mission  Need 
Statement,  and  the  Request  for  Proposal.  Of  those  sources,  only  the  Mission 
Need  Statement  is  a  source  document  for  user  requirements.  DoD 
Instruction  8120.2,  which  was  in  effect  at  the  time  of  the  SPS  program  initiation 
and  Milestone  I  review,  did  not  require  automated  information  systems  to  have 
an  Operations  Requirements  Document.^®  The  Operational  Requirements 
Docvunent  was  required  by  DoD  Instruction  5000.2  for  combat  systems.  The 
Operational  Requirements  Document  is  the  source  document  for  operational 


^Operational  requirements  as  referenced  in  this  report  refer  to  the  users' 
operational  performance  requirements  normally  stated  in  an  Operational 
Requirements  Document  or  Acquisition  Program  Baseline.  Those  requirements 
are  used  in  developmental  and  operational  testing.  Before  March  1996, 
automated  information  systems  were  not  required  to  have  an  Operational 
Requirements  Document. 

l^The  revised  DoD  Regulation  5000.2,  "Mandatory  Procedures  for  Major 
Defense  Acquisition  Programs  (MDAPs)  and  Major  Automated  Information 
Systems  (MAIS)  Acquisition  Programs,"  March  15,  1996,  requires  that  an 
Operational  Requirements  Document  be  prepared  for  Automated  Information 
Systems  at  Milestone  I. 
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requirements  used  in  determining  the  operational  suitability  and  effectiveness  of 
combat  systems  and  is  reflected  in  the  technical  and  operational  performance 
measures  in  the  TEMP. 

In  lieu  of  the  requirement  for  an  Operational  Requirements  Document,  DoD 
Instruction  8120.2  required  that  users  document  the  critical  operational  test 
criteria  that  will  be  used  as  the  basis  for  determining  the  effectiveness  and 
suitability  of  the  system.  Specifically,  DoD  Instruction  8120.2  required  that 
critical  operational  test  criteria  be  established  by  the  functional  user,  agreed  to 
by  the  lead  acquisition  authority,  and  documented  in  the  automated  information 
system  program  baseline,  in  accordance  with  DoD  7920.2-M,  "Automated 
Information  Systems  Life-Cycle  Management  Manual,"  March  1990. 

The  DoD  Test  Oversight  Officials  reviewed  the  SPS  TEMP,  dated 
October  20,  1995,  and  identified  a  need  for  user- validated  requirements.  On 
December  1,  1995,  the  Director,  Test  Systems  Engineering  and  Evaluation,  sent 
the  memorandum,  "Department  of  Defense  (DoD)  Standard  Procurement 
System  (SPS)  Milestone  I  Test  and  Evaluation  Master  Plan  (TEMP),  Final 
Coordination  Draft,"  October  20,  1995,  directing  that  the  users  validate  SPS 
user  requirements  before  TEMP  approval.  The  program  manager  submitted  a 
second  draft  TEMP,  dated  December  1,  1995.  That  TEMP  was  approved  by 
the  Director,  Operational  Test  and  Evaluation,  on  February  14,  1996,  to 
support  the  conduct  of  the  Functional  Capabilities  Demonstration.  This 
document  still  did  not  contain  the  user- validated  operational  requirements.  The 
Director,  Operational  Test  and  Evaluation,  directed  that  the  TEMP  be  updated 
to  include  those  operational  requirements  before  the  beginning  of 
demonstration,  validation,  and  acceptance  and  operational  testing. 

Effort  to  Identify  Requirements.  The  SPS  Test  and  Evaluation  Working 
Group  is  attempting  to  get  information  on  user  operational  requirements.  The 
SPS  test  director  developed  a  plan  for  going  directly  to  a  limited  number  of  user 
sites  to  collect  requirements  from  the  users.  The  SPS  program  manager  agreed 
to  fund  the  effort  in  the  interest  of  obtaining  data  needed  to  conduct  testing. 
Although  the  testers  and  user  representatives  work  together  in  developing  the 
technical  and  operational  performance  measures  reflected  in  the  TEMP,  the 
basis  for  developing  those  measures  are  the  users'  operational  performance 
requirements.  Those  operational  performance  requirements  should  be 
determined  by  the  users  independently  of  the  testers.  Because  the  Council 
represents  the  user  community,  we  believe  the  Council  should  provide  validated 
operational  performance  requirements  to  the  SPS  testers. 

Compressed  Schedule 


The  Integrated  Test  Program  Schedule  shown  in  the  December  1,  1995,  version 
of  the  TEMP  does  not  allow  sufficient  time  for  comprehensive  test  planning 
between  the  identification  of  the  first  increment  of  the  SPS  at  contract  award 
and  the  initiation  of  testing.  Table  2  depicts  the  key  test  events  and  schedule. 
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Table  2.  Program  Events  Affecting  SPS  Testing 
(as  of  December  1,  1995) 

_ Event _  _ Schedule 


Increment  One  Functional 
Capabilities  Demonstration 

First  stage 

Developmental  Test  and  Evaluation 

SPS  contract  award 
(One  or  more  contractors) 

Increment  One  Validation  and 
Acceptance  Testing 

First  stage 

Initial  Operational  Test  and  Evaluation 
Down  select  to  one  contractor 


Increment  One  Initial  Operational 
Test  and  Evaluation 


SPS  Program  Milestone  II/IIIA 
Review  (increment  one) 


Increment  One  deployment 


Source:  Draft  Test  and  Evaluation  Master 


November  15,  1995,  through  March  15,  1996 
Last  Stage  Functional  Capabilities  Demonstration 
May  15,  1996 

June  17,  1996,  through  September  30,  1996 

Last  stage  Validation  and  Acceptance  Testing 
December  15,  1996 

January  6,  1997,  through  February  21,  1997 
March  31,  1997 

May  15,  1997,  through  May  14,  1998 
Plan,  December  1,  1995. 


Development  Test  and  Evaluation  Schedule.  The  testers  will  not  have  the 
complete  list  of  functions  of  the  SPS  for  the  first  increment  until  contract 
award,  which  is  about  1  month  before  the  beginning  of  formal  developmental 
testing.  Increment  One  consists  of  the  functionality  that  is  identified  in  the 
offeror's  proposal  for  the  Functional  Capabilities  Demonstration  phase. 

To  mitigate  risk  associated  with  test  planning  under  a  compressed  program 
schedule,  the  testers  scheduled  the  initial  phase  of  developmental  testing  for 
Increment  One  several  months  before  contract  award.  That  initial  phase  of 
developmental  testing  will  occur  during  the  last  portion  of  the  Functional 
Capabilities  Demonstration.  However,  the  final  configuration  of  functions  to  be 
available  from  the  vendors  at  the  Functional  Capabilities  Demonstration  will  not 
be  known  until  the  contract  is  awarded  to  one  or  more  vendors.  In  our  opinion, 
1  month  between  the  identification  of  the  functions  for  the  first  increment  for 
the  SPS  at  contract  award  and  the  beginning  of  the  developmental  testing  is  not 
sufficient  to  ensure  that  the  test  plan  will  reflect  the  full  range  of  technical 
requirements. 

Operational  Test  and  Evaluation  Schedule.  We  believe  that  the  Integrated 
Test  Program  Schedule  in  the  December  1,  1995,  TEMP  does  not  allow 
sufficient  time  between  contract  award  and  the  beginning  of  operational  testing 
and  evaluation  for  the  operational  testers  to  comply  with  DoD  Instruction 
5000.2  requirements  aimed  at  ensuring  that  testing  is  adequate.  DoD 
Instruction  5000.2  requires  the  DoD  Components  to  brief  the  Director, 
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Operational  Test  and  Evaluation,  on  the  concepts  for  testing  and  evaluation 
120  days  before  testing  begins  and  to  submit  the  test  plan  to  the  Director, 
Operational  Test  and  Evaluation,  60  days  before  the  test.“ 

The  December  1,  1995,  schedule  requires  beginning  the  first  stage  of 
operational  test  and  evaluation  (the  Operational  Assessment)  during  the 
developmental  test  phase  (Validation  and  Acceptance  Testing)/^  The  SPS 
program  manager  scheduled  the  Validation  and  Acceptance  Testing  to  begin 
about  1  month  after  contract  award.  Part  IV  of  the  TEMP,  Operational  Test 
and  Evaluation  Outline,  states  that  "The  OT&E  [Operational  Test  and 
Evaluation]  for  increment  one  will  be  defined  no  later  than  initiation  of 
Increment  One  Demonstration/Validation  Testing."  That  schedule  would 
require  that  the  operational  testers  brief  the  Director,  Operational  Test  and 
Evaluation,  on  the  test  and  evaluation  concept  before  the  completion  of  the 
Functional  Capabilities  Demonstration  phase.  The  schedule  would  also  require 
that  the  operational  testers  submit  a  copy  of  the  detailed  test  plan  to  the 
Director,  Operational  Test  and  Evaluation  about  1  month  before  contract  award. 
The  functions  for  the  first  increment  of  the  SPS  will  not  be  known  until  contract 
award.  Accordingly,  we  believe  that  the  operational  test  plan  may  not  be 
comprehensive  enough  to  eliminate  the  considerable  risk  that  Ae  first  increment 
will  not  be  tested  well  enough  to  determine  whether  the  SPS  is  operationally 
suitable  and  effective. 


Testing  for  CapabiKty  to  Share  Data 


The  SPS  TEMP  does  not  define  testing  for  the  Shared  Data  Warehouse  on  the 
Integrated  Program  Schedule  or  in  the  discussion  of  plaimed  developmental  or 
operational  testing.  Although  the  Shared  Data  Warehouse  is  an  integral  part  of 
achieving  the  full  capabilities  of  SPS,  the  program  strategy  does  not  define  the 
requirements  for  the  first  functional  increment  or  subsequent  increments. 

Multiple  Current  and  Evolving  Systems.  The  shared  data  requirements 
wifliin  die  DoD  include  the  Corporate  Information  Management  standard 
systems  (for  example  finance,  accounting,  and  logistics  systems);  specific 
Military  Department  systems;  the  evolving  Defense  Information  Systems 


llThe  requirement  for  the  Components  to  brief  120  days  before  testing  and  to 
submit  a  test  plan  60  days  before  testing  is  also  in  revised  DoD 
Regulation  5000.2,  "Mandatory  Procedures  for  Major  Defense  Acquisition 
Programs  (MDAPs)  and  Major  Automated  Information  Systems  (MAIS) 
Acquisition  Programs,"  March  15,  1996. 

l^DemonstrationA^alidation  Testing  is  used  interchangeably  with  the  term 
Demonstration,  Validation,  and  Acceptance  Testing  in  the  TEMP  to  refer  to  the 
developmental  testing  phase. 
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Network;  the  evolving  Defense  Messaging  Service;  the  existing  DoD  Electronic 
Commerce/Electronic  Data  Interchange  standard  architecture;  and  the  evolving 
Defense  Data  Repository  .  In  addition,  the  SPS  is  planned  to  share  data  wift 
many  systems  outside  the  DoD  to  include  the  evolving  Federal  Central 
Contractor  Registration  System  and  the  evolving  Federal  Acquisition  Computer 
Network.  However,  the  functional  user  has  not  fully  defined  the  role  of  the 
Shared  Data  Warehouse  in  providing  an  information  gateway  to  other  DoD 
automated  information  systems  and  those  outside  the  DoD. 

Testing  Strategy  for  Shared  Data.  The  SPS  TEMP  does  not  identify  the 
testing  needed  to  ensure  that  the  requirement  for  the  Shared  Data  Warehouse  is 
met  and  does  not  identify  a  strategy  to  ensure  that  the  requirement  is  met  in  the 
future.  The  developmental  and  operational  tester  cannot  develop  a 
comprehensive  test  plan  to  ensure  that  die  SPS  achieves  the  goal  of  a  Shared 
Data  Warehouse  until  the  Council  and  the  SPS  program  manager  define  the 
requirements  and  how  to  achieve  them. 


Conclusion 


The  Council  must  develop  and  validate  operational  user  requirements  for  the 
SPS  to  ensure  testing  adequacy.  Without  user- validated  operational 
requirements,  significant  risk  develops  for  inadequate  testing  and  for  accepting  a 
system  that  will  not  meet  the  users'  needs.  In  addition,  the  compressed  time 
schedule  in  the  Integrated  Test  Program  Schedule  does  not  allow  enough  time  to 
ensure  that  testers  can  develop  a  comprehensive  test  plan  that  determines 
whether  SPS  fully  meets  user-defined  operational  performance  requirements. 
The  SPS  will  not  be  fully  functional  until  the  Shared  Data  Warehouse  is 
achieved.  The  Council,  as  the  functional  user  representative,  must  define 
requirements  for  the  Shared  Data  Warehouse.  The  program  manager  can  then 
develop  the  testing  strategy  for  data  sharing. 


Recommendations,  Management  Comments,  and  Audit 
Response 


Revised  Recommendation.  As  a  result  of  management  comments,  we  revised 
Recommendations  B.l.a.  and  B.l.c.  to  specify  which  requirements  need  to  be 
identified  and  validated  and  to  establish  a  schedule  for  identifying  Shared  Data 
Warehouse  requirements,  respectively. 

B.l.  We  recommend  that  the  Director,  Defense  Procurement,  direct  the 
Procurement  Corporate  Information  Management  Council  as  the  functional 
user  representative  to: 
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a.  Develop  and  validate  the  operational  performance  requirements 
for  the  Standard  Procurement  System. 

b.  Provide  the  operational  performance  requirements  to  the 
Director,  Defense  Procurement  Corporate  Information  Management 
Systems  Center,  Defense  Logistics  Agency,  for  inclusion  in  the  Standard 
Procurement  System  Integrated  Test  Program  Schedule. 

c.  In  coordination  with  the  Standard  Procurement  System  Program 
Management  OfHce,  establish  a  schedule  to  define  the  requirements  for  the 
shared  data  base  portion  of  the  Standard  Procurement  System  and 
incorporate  them  into  the  functional  requirement,  Mission  Needs 
Statement,  and  Operational  Requirements  Document  and  provide  them  to 
the  program  manager. 

Management  Comments.  The  Director,  Defense  Procurement,  stated  that 
Recommendation  B.l.a.  had  been  accomplished  because  the  SPS  solicitation 
had  been  amended  to  more  clearly  identify  validated  operational  environments 
in  which  the  SPS  must  operate  and  that  the  solicitation  identifies  the  functions 
required  of  SPS.  The  Director  concurred  with  Recommendation  B.l.b.,  stating 
that  the  SPS  Operational  Requirements  Document  is  being  developed  and  will 
be  available  for  Initial  Operational  Test  and  Evaluation,  presently  scheduled  for 
February  24,  1997.  The  Director  partially  concurred  with  Recommendation 
B.I.C.,  stating  that  basic  data  warehousing  requirements  had  been  provided  to 
the  SPS  Program  Manager.  However,  more  detailed  requirements  cannot 
currently  be  provided  because  factors  beyond  the  Director's  control  will  impact 
those  requirements,  such  as  changes  to  law  or  regulation  and  changes  in  user 
needs  for  data  warehousing. 

Audit  Response.  Because  the  draft  recommendation  was  not  well-phrased,  the 
Director's  comments  regarding  Recommendation  B.l.a.  does  not  address  our 
primary  concern.  That  concern  is  witii  the  development  and  validation  of  SPS 
operational  performance  requirements,  which  should  be  in  the  Operational 
Requirements  Document.  Operational  performance  requirements  are  critical  to 
the  development  of  effective  test  criteria.  We  revised  the  recommendation  to 
more  clearly  express  our  intent  and  ask  that  the  Director,  Defense  Procurement, 
provide  conunents  on  the  revised  recommendation  in  response  to  the  final 
report. 

The  Director's  comments  on  Recommendation  B.l.b  were  fully  response  and  no 
further  conmient  is  required. 

Regarding  the  Director's  response  to  Recommendation  B.l.c,  we  understand 
that  firm,  detailed  user  requirements  of  the  Shared  Data  Warehouse  cannot  be 
provided  at  this  time.  However,  as  the  user  representative  for  the  DoD 
procurement  community,  providing  those  requirements  is  a  Council 
responsibility.  Accordingly,  we  revised  Recommendation  B.l.c.  to  help  enable 
that  responsibility  to  be  met  in  a  timely  and  effective  manner.  We  ask  that  the 
Director  provide  comments  on  the  revised  recommendation  in  response  to  the 
final  report. 
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B.2.  We  recommend  that  the  Director,  Defense  Procurement  Corporate 
Information  Management  Systems  Center,  Defense  Logistics  Agency: 

a.  Establish  an  Integrated  Test  Program  Schedule  that  allows 
sufficient  time  for  detailed  test  planning  and  the  required  brieHugs  and 
review  by  the  Director  of  Operational  Test  and  Evaluation. 

b.  Develop  the  testing  strategy  for  shared  data  based  on  the 
requirements  deflnition  provided  by  the  Procurement  Corporate 
Information  Management  Council. 

Defense  Logistics  Agency  Comments.  The  Defense  Logistics  Agency  partially 
concurred  with  Recommendation  B.2.a.,  stating  that  the  SPS  Integrated  Test 
Program  Schedule  provides  sufficient  time  for  test  planning  and  necessary 
reviews.  However,  the  Agency  would  make  any  changes  deemed  necessary  to 
that  schedule  by  the  SPS  Test  Integrated  Product  Team.  The  Agency  concurred 
with  Recommendation  B.2.b.  and  stated  that  the  testing  strategy  for  the  SPS 
Shared  Data  Warehouse  would  be  developed  and  integrated  into  the  overall  SPS 
testing  plans  by  April  1999. 

Audit  Response.  Although  the  Agency  did  not  fully  concur  with 
Recommendation  B.2.a.,  its  response  meets  the  intent  of  that  recommendation. 
Agency  comments  regarding  Recommendation  B.2.b.  are  fiilly  responsive. 
Accordingly,  no  further  Agency  comment  is  required  . 


24 


Part  n  -  Additional  Information 


Appendix  A.  Audit  Process 


Scope  and  Methodology 


Allegations  to  DoD  Hotline.  To  determine  the  validity  of  the  allegations,  we 
examined  the  system  requirements  in  the  functional  description  of  the  SPS  draft 
and  final  solicitations,  dated  November  1994  through  March  1996.  We  also 
reviewed  program  management  office  documentation  related  to  the  SPS 
acquisition  planning  and  execution,  cost-benefit  analyses,  and  test  and 
evaluation  and  discussed  those  subjects  with  SPS  program  management 
personnel.  We  interviewed  personnel  who  provided  oversight  of  the  SPS 
program  from  the  offices  of  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology;  Under  Secretary  of  Defense  (Comptroller);  and  Assistant  Secretary 
of  Defense  (Command,  Control,  Communications,  and  Intelligence).  We  also 
interviewed  members  of  the  Procurement  CIM  Council  and  SPS  Source 
Selection  Advisoiy  Committee.  We  did  not  rely  on  computer-processed  data  or 
statistical  sampling  procedures  during  the  audit.  We  included  tests  of 
management  controls  considered  necessary. 

Audit  Period,  Standards,  and  Location.  We  performed  this  economy  and 
efficiency  audit  from  September  1995  through  April  1996  in  accordance  with 
auditing  standards  issued  by  the  Comptroller  General  of  the  United  States,  as 
implemented  by  the  Inspector  General,  DoD.  The  audit  was  primarily  made  at 
the  Defense  Logistics  Agency,  Fort  Belvoir,  Virginia.  Appendix  F  lists  the 
organizations  we  visited  or  contacted  during  the  audit. 


Management  Control  Program 


DoD  Directive  5010.38,  "Internal  Management  Control  Program,"  April  14, 
1987,  requires  DoD  organizations  to  implement  a  comprehensive  system  of 
management  controls  that  provides  reasonable  assurance  that  programs  are 
operating  as  intended  and  to  evaluate  the  adequacy  of  the  controls. 

Scope  of  Review  of  Management  Control  Program.  We  evaluated  DPCSC 
management  controls  over  the  planning,  development,  and  execution  of  the  SPS 
program.  We  focused  on  those  controls  applicable  to  the  allegations  submitted 
to  the  Defense  Hotline.  We  did  not  review  management's  self-evaluation  of  the 
applicable  controls  because  management  did  not  perform  a  formal  self- 
assessment. 

Adequacy  of  Management  Controls.  We  identifred  a  material  management 
control  weakness  in  &at  the  DPCSC  did  not  establish  policies  and  procedures  to 
implement  a  comprehensive  management  control  program  for  FY  1995. 
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However,  during  the  audit,  the  DPCSC  began  restructuring  internal  policies  and 
procedures  to  fully  implement  a  management  control  program,  "nie  DPCSC 
Director  stated  that  tite  management  control  program  would  be  in  full 
compliance  with  DoD  Directive  5010.38  by  May  31,  1996.  We  consider 
management  actions  to  be  appropriate.  Accordingly,  we  made  no 
recommendations  regarding  the  management  control  program  at  the  DPCSC. 

Adequacy  of  Management's  Self-Evaluation.  Defense  Logistics  Agency 
officials  did  not  identify  the  DPCSC  as  an  assessable  unit  and,  therefore,  did 
not  detect  or  report  the  material  management  control  weakness  identified  by  the 
audit.  Although  the  DPCSC  did  not  conduct  a  formal  self-evaluation  the 
DPCSC  identified  some  of  its  susceptibility  to  fraud,  waste,  abuse,  and 
mismanagement  in  the  May  1995  SPS  Risk  Management  Plan,  which  was 
prepared  as  part  of  the  SPS  Program  Management  Plan.  The  1995  SPS  Risk 
Management  Plan  did  not  identify  the  specific  material  management  control 
weakness  identified  by  the  audit  because  that  plan  focused  on  external  risk 
events  that  could  delay  the  development  and  deployment  of  the  SPS. 
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The  SPS  was  not  the  subject  of  any  previous  audit  coverage;  however,  the  DoD 
published  one  technical  report  and  one  study  that  directly  related  to  the  SPS 
acquisition  strategy  to  acquire  and  modify  a  commercial  off-the-shelf  software 
package. 


Department  of  the  Air  Force 


The  Department  of  the  Air  Force  Software  Technology  Support  Center  issued 
"Guidelines  for  Successful  Acquisition  and  Management  of  Software  Intensive 
Systems:  Weapon  Systems;  Command  and  Control  Systems;  and  Management 
Information  Systems,"  in  February  1995.  Those  guidelines  recommended  that 
DoD  organizations  planning  to  acquire  conunercial  off-the-shelf  software 
packages  do  so  with  no  intentions  of  altering  the  software  packages.  According 
to  the  guidelines,  purchasers  of  commercial  off-the-shelf  packages  should  not 
alter  the  software  and  should  avoid  having  the  software  altered  in  order  for  the 
software  to  be  compatible  with  subsequent  commercial  versions. 

The  guidelines  did  not  contain  specific  recommendations,  but  instead  listed 
several  important  commercial  software  products  "facts  of  life,"  which  should  be 
used  in  decisionmaking  for  large  enterprise  systems.  The  "facts  of  life" 
included: 

0  conunercial  products  are  not  interoperable  with  other  conunercial 
products, 

o  product  literature  overstates  software  capability, 
o  products  never  exactly  match  their  users'  needs, 

0  unique  commercial  off-the-shelf  versions  are  costly,  and 
o  upgrades  are  frequent  and  asynchronous. 


Defense  Science  Board 


The  Defense  Science  Board  Task  Force  issued  its  report  on  "Acquiring  Defense 
Software  Conunercially"  to  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  in  June  1994.  The  task  force  was  to  determine  the  "conditions 
under  which  procurement  of  defense  software  can  use  commercial  practices" 
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and  to  define  the  "changes  required  to  permit  such  use."  The  Task  Force 
concluded  that  the  DoD  needed  a  more  coordinated  approach  to  oversight  of 
diverse  software  capabilities  and  programs. 

The  Task  Force  report  stated  that  DoD  had  not  determined  when  to  use 
commercial  off-the-shelf  software  packages  and  that  if  a  commercial  software 
package  is  acquired,  it  should  be  used  as  is,  without  modification.  The  Task 
Force  recommended  that  DoD  identify  the  advantages  and  disadvantages  of 
using  commercial  software  and  establish  requirements  for  software 
architectures. 
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We  noted  that  current  DoD  acquisition  objectives  and  procedures  do  not 
accommodate  the  SPS  strategy  of  acquiring  a  commercial  software  system.  We 
also  noted  a  unique  organizational  structure  for  two  SPS  advisory  councils. 

Acquisition  Program  Baseline.  DoD  Regulation  5000.2-R,  "Mandatory 
Procedures  for  Major  Defense  Acquisition  Programs  (MDAPs)  and  Major 
Automated  Information  System  (MAIS)  Acquisition  Programs," 
March  15,  1996,  requires  DoD  acquisition  programs  to  establish  cost,  schedule, 
and  performance  goals  at  program  initiation.  Those  goals  should  be  formally  in 
an  acquisition  program  baseline  and  should  be  stated  in  terms  of  threshold 
(minimum  acceptable)  and  objective  (desired)  values.  The  differences  between 
objective  and  tlureshold  values  represent  the  degree  to  which  "trade-offs"  can 
occur  between  cost,  schedule,  or  performance  goals.  Trade-off  decisions  made 
early  in  the  acquisition  process  can  reduce  total  life-cycle  costs. 

The  SPS  program  manager  had  not  established  an  acquisition  program  baseline. 
The  SPS  acquisition  approach  deterred  the  development  of  an  acquisition 
program  baseline  or  a  borough  evaluation  of  potential  trade-offs  early  in  the 
SPS  acquisition  cycle.  Because  the  basic  SPS  acquisition  strategy  is  to  initially 
acquire  an  existing  commercial  system  that  best  meets  overall  DoD 
requirements,  SPS  program  cost,  schedule,  and  performance  factors  will  be 
largely  dependent  on  the  negotiated  contract  terms  and  specific  capabilities  of 
the  system  eventually  selected.  Accordingly,  the  SPS  program  manager  will  not 
have  the  information  necessary  to  effectively  formulate  an  acquisition  program 
baseline  or  to  perform  meaningful  trade-off  analyses.  That  information  will  not 
be  sufficiently  defined  until  contract  award,  at  best,  or  until  the  final  SPS 
product  is  determined  after  selection  of  the  successful  vendor. 

SPS  Advisory  Councils.  The  Goldwater-Nichols  Department  of  Defense 
Reorganization  Act  of  1986  established  a  new  reporting  chain  for  DoD 
acquisition  officials  to  clearly  separate  the  requesting  (user)  community  from  the 
acquisition  (procurement)  organization.  Congress  established  that  Act  to  deter 
undue  influence  by  the  user  community  on  the  acquisition  process.  We  noted 
that  two  SPS  advisory  councils  were  primarily  composed  of  the  same 
individuals,  even  though  the  councils  have  separate  and  distinct  purposes. 

The  primary  purpose  of  the  Procurement  CIM  Council  is  to  provide  user 
requirements  and  perspective  to  the  SPS  program  manager.  Conversely,  the 
main  purpose  of  the  Source  Selection  Advisory  Council  is  to  evaluate  various 
procurement  options  and  procedures  and  make  related  recommendations  to  the 
Source  Selection  Authority.  Besides  having  a  common  majority  membership, 
both  councils  are  chaired  by  the  same  individual.  SPS  program  management 
officials  stated  that  it  was  sometimes  unclear  under  which  authority  the  advisory 
officials  provided  guidance  and  whether  the  guidance  was  appropriate.  That 
confusion  was  compounded  because  the  two  councils  did  not  routinely  or  fully 
document  proceedings.  The  lack  of  documentation  also  contributed  to  a  lack  of 
accountability.  However,  we  found  nothing  to  prohibit  both  councils  from 
having  a  common  membership. 
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From  April  through  October  1995,  four  draft  SPS  solicitations  were  issued  for 
comment;  the  final  solicitation  was  issued  on  October  30,  1995.  On 
July  31,  1995,  10  days  after  the  second  draft  was  issued,  the  complainant 
submitted  allegations  to  the  Defense  Hotline.  Those  allegations  expressed 
concerns  about  the  SPS  in  three  areas:  functional  requirements,  acquisition 
strategy,  and  testing  plans.  The  complainant  chose  to  remain  anonymous. 
Accordingly,  we  interpreted  and  evaluated  the  allegations  based  on 
documentation  and  other  information  the  complainant  may  or  may  not  have 
reviewed.  The  complaint  contained  11  allegations.  We  grouped  the  allegations 
into  the  three  areas  of  concern.  Further,  we  separated  some  allegations  into 
multiple  parts.  Accordingly,  we  evaluated  13  distinct  allegations.  The  three 
general  areas  of  concern,  the  13  allegations,  and  the  related  audit  results  follow. 

Concern  1:  The  strategy  to  acquire  and  deploy  SPS  is  flawed. 

Audit  Result:  The  concern  was  partially  substantiated  because  the  SPS  will 
initially  be  deployed  to  organizations  having  manual  and  semi-automated 
systems  instead  of  to  large  volume  users  or  to  the  Defense  Contract 
Management  Command.  According  to  the  SPS  program  manager,  that 
deployment  strategy  will  provide  the  greatest  return  on  investment  to  die  DoD. 
However,  the  costs  and  benefits  of  deployment  alternatives  were  not  analyzed. 
See  Finding  A  for  a  complete  discussion  on  the  strategy  to  acquire  SPS. 

a.  The  SPS  is  being  designed  and  developed  to  initially  satisfy  only 
"small  dollar,  small  purchase"  organizations.  Based  on  that  limited  scope, 
an  incomplete  product  may  be  developed  and  delivered,  requiring  increased 
expense  and  effort  to  achieve  the  goal  of  a  shared  data  base. 

Audit  Result.  The  allegation  was  partially  substantiated,  but  was  the  result  of  a 
conscious  decision.  The  program  manager  decided  to  initially  deploy  the  SPS 
to  procurement  organizations  using  manual  or  semi-automated  systems.  Those 
organizations  include  most  small  purchase  or  small  volume  users,  but  also 
include  some  large,  high- volume  procurement  organizations,  such  as 
Headquarters,  Space  and  Naval  Warfare  Systems  Command.  The  SPS  program 
manager  clearly  expects  to  deploy  subsequent  increments  of  the  SPS  to 
organizations  that  have  more  complex  requirements.  However,  the  SPS 
program  manager  did  not  perform  a  cost-benefits  analysis  of  the  different 
approaches  considered  for  the  development  and  deployment  of  SPS. 

b.  Defense  Contract  Management  Command  users  will  not  get  SPS 
software  until  late  in  the  system  development  cycle.  Accordingly, 
concurrent  operations  of  the  SPS  and  the  Mechanization  of  Contract 
Administration  Services  System,  which  the  Defense  Contract  Management 
Command  uses,  will  be  required,  but  will  not  be  cost-effective.  That  major 
change  in  acquisition  approach  is  contrary  to  the  approach  described  to 
conunand  users. 
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Audit  Result,  "nie  allegation  was  partially  substantiated,  but  was  the  result  of  a 
deliberate  decision.  According  to  the  SPS  deployment  plans,  which  the 
MAISRC  reviewed  and  approved,  the  Defense  Contract  Management  Command 
will  be  one  of  the  last  organizations  to  receive  SPS.  One  factor  considered  is 
that  the  Defense  Contract  Management  Command  has  the  most  automated 
capability  of  any  SPS-targeted  user.  However,  the  decision  to  delay 
deployment  until  late  in  the  system  development  cycle  will  require  the  longest 
period  of  concurrent  operations  by  the  SPS  and  the  Mechanization  of  Contract 
Administration  Services  System.  Although  the  SPS  development  and 
deployment  approach  has  several  advantages,  its  cost-effectiveness  has  not  been 
documented.  We  found  no  indication  that  the  SPS  program  manager  advocated 
any  other  deployment  approach. 

c.  The  SPS  may  be  developed  differently  for  each  customer  base  in 
order  to  incorporate  unique  requirements. 

Audit  Result.  The  allegation  was  not  substantiated.  According  to  the  program 
manager,  the  SPS  will  be  developed  incrementally  based  on  industry's  ability  to 
meet  overall  DoD  requirements  and  will  not  be  based  on  the  particular  needs  of 
any  user.  We  found  no  evidence  that  different  versions  of  the  SPS  would  be 
developed  for  different  users.  The  solicitation  requires  that  the  SPS  be 
sufficiently  flexible  to  establish  and  display  contract  information  according  to 
users'  functions  and  to  incorporate  DoD  Component  or  organization-unique 
procedures. 

Concern  2:  The  SPS  will  not  meet  some  specific  functional  requirements. 

Audit  Results.  The  overall  concern  generally  had  no  merit.  Two  specific 
functional  requirements  allegedly  omitted  were  not  in  any  draft  solicitation  or 
the  final  solicitation.  However,  these  capabilities  could  be  provided  through 
other  functional  requirements  in  the  final  solicitation.  The  remaining  functional 
requirements  allegedly  omitted  were  specifically  in  draft  and  final  solicitations. 

a.  The  SPS  will  not  provide  the  "alerts  that  are  required  to  assure 
that  the  contractor  is  in  compliance  with  the  contract  at  the  least  cost  to  the 
Grovernment." 

Audit  Result.  The  allegation  was  not  substantiated.  Each  draft  solicitation  and 
the  final  solicitation  included  specific  requirements  to  automatically  alert 
appropriate  users  when  contractor  performance  exceeded  the  pre-defined 
criteria. 

b.  The  first  release  of  the  SPS  does  not  include  requirements  for 
"closeout  of  contracts." 

Audit  Result.  The  allegation  was  not  substantiated.  The  July  1995  draft 
solicitation  and  each  subsequent  solicitation  included  contract  close-out 
requirements  for  the  SPS. 
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c.  The  new  system  will  not  validate  the  "long  line  of  accounting 

data." 

Audit  Result.  The  allegation  was  not  substantiated.  Each  draft  solicitation  and 
the  final  solicitation  included  specific  requirements  to  validate  accounting 
classification  data  and  to  reject  invalid  data, 

d.  The  SPS  does  not  include  requirements  for  "[on-line]  help 
messages." 

Audit  Result.  The  allegation  was  not  substantiated.  The  July  1995  draft 
solicitation  and  each  subsequent  solicitation  include  specific  requirements  for 
on-screen  help  messages  and  tutorials. 

e.  The  SPS  does  not  include  requirements  for  "multiple  payment 
offices." 

Audit  Result.  The  allegation  was  substantiated;  none  of  the  SPS  draft 
solicitations  or  the  final  solicitation  included  a  specific  requirement  for 
accommodating  multiple  payment  offices.  However,  the  final  solicitation 
requires  that  SPS  have  templates  that  permit  the  user  to  describe  and  define 
user-created  data  elements.  Accordingly,  multiple  payment  offices,  or  any 
other  needed  data  elements,  could  be  designated  and  incorporated  within  the 
SPS  as  required. 

f.  The  SPS  does  not  include  requirements  for  "currencies  other  than 
U.S.  dollars." 

Audit  Result.  The  allegation  was  not  substantiated.  The  July  1995  draft 
solicitation  and  each  subsequent  solicitation  included  specific  requirements  to 
allow  unit  prices  and  awards  in  currencies  other  than  U.S.  dollars. 

g.  The  SPS  does  not  include  requirements  to  do  multiple  actions. 
For  example,  a  modification  adds  money  and  a  line  item  and  changes  the 
schedule  for  that  one  line  item.  That  modification  would  take  at  least  three 
changes  using  the  SPS,  but  the  Mechanization  of  Contract  Administration 
Services  System  makes  the  modification  with  only  one  change  with  multiple 
screens. 

Audit  Result.  The  allegation  was  substantiated.  Neither  the  draft  solicitation 
nor  the  final  solicitation  specified  a  requirement  for  automatically  performing 
multiple  actions.  However,  the  final  solicitation  requires  a  capability  for 
implementing  user-defined  criteria.  User-defined  criteria  could  be  used  to 
designate  the  automatic  linkage  of  actions  associated  with  a  particular  function. 
Accordingly,  the  actions  associated  with  contract  modifications  could  be  defined 
by  the  user  and  could  be  automatically  executed  whenever  a  particular  contract 
is  modified. 
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h.  "It  has  been  said  that  the  Military  Standard  Contract 
Administration  Procedures  requirement  may  be  scrapped  because  it  is  so 
difHcult  to  program." 

Audit  Result.  The  allegation  was  not  substantiated.  Each  draft  solicitation  and 
the  final  solicitation  included  specific  requirements  to  generate,  receive, 
validate,  and  record  transaction  sets  using  Military  Standard  Contract 
Administration  Procedures. 

Concern  3:  Specific  aspects  of  the  SPS  testing  plans  are  inadequate. 

Audit  Result.  The  concern  had  limited  merit  because  the  December  1,  1995, 
TEMP  did  not  include  all  parties  potentially  affected  throughout  the  SPS  testing 
process.  Other  aspects  of  the  SPS  testing  plans  are  discussed  in  Finding  B. 

a.  Early  versions  of  the  SPS  "will  not  be  tested  for  stress." 

Audit  Result.  The  allegation  was  substantiated;  the  Draft  TEMP,  dated  May  8, 
1995,  made  no  provision  for  stress  testing.  However,  the  December  1,  1995, 
draft  TEMP  clearly  indicates  that  SPS  stress  testing  is  planned.  Although  the 
description  in  the  December  1,  1995,  TEMP  for  the  Initial  Operational  Test  and 
Evaluation  scenarios  were  not  detailed,  the  description  specifically  stated  that 
scenarios  would  include  operations  under  surge  conditions.  The  TEMP  also 
states  that  testing  of  follow-on  SPS  increments  will  mirror  those  scenarios. 

b.  Defense  Contract  Management  Command  and  Defense  Finance 
and  Accounting  Service  representatives  will  not  be  invited  to  all  SPS 
testing;  therefore,  testing  may  not  include  necessary  users. 

Audit  Result.  The  allegation  was  partially  substantiated.  The  TEMP  dated 
December  1,  1995,  did  not  state  that  Defense  Finance  and  Accounting  Service 
personnel  would  be  involved  in  any  SPS  testing.  However,  the  Defense 
Finance  and  Accounting  Service  will  have  no  critical  interest  in  the  SPS  until 
the  later  increments.  Further,  a  Defense  Finance  and  Accounting  Service 
representative  is  assigned  to  the  DPCSC.  The  TEMP  indicates  that  the  Defense 
Contract  Management  Command  will  be  involved  in  the  operational  testing  of 
the  SPS. 
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Management  Comments  on  Finding  A 


The  Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 
Intelligence)  generally  concurred  with  Finding  A,  "Strategy  for  Acquiring  the 
Standard  Procurement  System,"  but  provided  no  specific  comments  on  the 
finding  or  discussion.  The  Director,  Defense  Procurement,  stated  that  the 
report's  findings  and  recommendations  should  be  re-evaluated  because  the  SPS 
solicitation  was  amended  after  the  audit  was  completed.  Additionally,  the 
Director's  comments,  which  discussed  several  topics  in  the  report,  indicated 
disagreement  with  Finding  A.  The  Defense  Logistics  Agency  partially 
concurred  with  Finding  A,  but  nonconcurred  with  many  of  the  topics  discussed. 
The  following  summarizes  management  comments  concerning  the  draft  Finding 
A  and  provides  the  audit  response. 

Management  Comments  on  Finding  A,  "Strategy  for  Acquiring  the  SPS." 
The  Defense  Logistics  Agency  (the  Agency)  stated  that  describing  fee  SPS 
Program  as  a  $4.1  billion  program  was  misleading  and  incorrect  because  that 
amount  is  based  on  a  period  5  years  longer  than  the  SPS  Program's  approved 
length  and  includes  costs  neither  incurred  nor  controlled  by  the  SPS  Program 
Manager.  Further,  the  $4.1  billion  was  cited  merely  to  capture  the  attention  of 
potential  readers.  Additionally,  the  finding  did  not  reflect  potential  costs  of 
$8.1  billion  which  DoD  would  risk  by  doing  nothing  to  improve  automated 
support  to  the  procurement  community. 

The  Agency  did  not  agree  that  the  risk  factors  associated  with  the  SPS 
acquisition  strategy  may  result  in  unmet  user  needs  or  unanticipated  contract 
costs.  The  Agency  stated  that  3  years  of  effort  was  put  forth  to  identify 
standard  processes  and  procedures  for  future  use  by  the  procurement  community 
and  that  the  SPS  will  provide  those  standards  to  its  users.  Additionally,  great 
effort  had  been  taken  to  ensure  that  SPS  cost  projections  were  realistic. 
Offerors  have  demonstrated  a  good  understanding  of  SPS  requirements  and  feel 
confident  that  those  requirements  can  be  costed  on  a  fixed  price  basis  in  the  best 
and  final  offers.  Additionally,  the  Agency  stated  that  the  report  does  not 
adequately  recognize  the  role  of  SPS  Program  oversight  bodies.  In  accordance 
with  DoD  policy,  execution  of  the  SPS  Program  is  monitored  by  both  the 
Agency  and  Office  of  the  Secretary  of  Defense  and  any  unanticipated  cost 
growth  would  be  quickly  identified  and  appropriately  dealt  with. 

The  Agency  could  neither  understand  nor  support  the  position  that  a  commercial 
acquisition  strategy  adds  risk  to  the  SPS  program.  The  Agency  also  cited 
several  Federal  Government-  and  DoD-stated  preferences  for  acquiring 
commercial  products  over  the  in-house  development  of  unique  products. 
Further,  the  contracting  method  being  used  to  acquire  SPS  is  fair  and  open 
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competition  to  purchase  a  commercially  available  system.  Because  a 
commercially  available  system  is  being  acquired,  fixed  prices  are  required  by 
FASA  and  the  Federal  Acquisition  Regulation. 

Audit  Response.  The  estimated  SPS  life-cycle  cost,  as  shown  in  the  SPS 
Economic  Analysis  prepared  by  the  SPS  program  management  office  and 
submitted  to  the  MAISRC  to  support  the  SPS  Milestone  I  decision,  was 
estimated  at  $4.1  billion.  Further,  that  amount  was  based  on  a  program  life  of 
15  years  and,  in  accordance  with  the  definition  of  life-cycle  cost,  included  all 
associated  costs,  including  the  cost  of  operations.  Fu^er,  we  specify  SPS 
program  costs  in  the  report's  background.  However,  we  have  clarified  the 
finding  to  indicate  that  the  $4.1  billion  was  an  estimated  life-cycle  cost  to  avoid 
any  confusion. 

Contrary  to  Agency  comments  and  inference,  we  are  not  opposed  to  the  DoD 
acquisition  and  use  of  existing,  commercially  available  products  when  those 
products  will  meet  DoD  needs.  In  fact,  we  have  long  advocated  the  use  of 
commercial,  off-the-shelf  automation  resources  when  appropriate.  For  instance, 
in  Inspector  General,  DoD,  Report  91-121,  "Use  of  Mobile 
Computers  -  Army,"  September  23,  1991,  we  recommended  that  the  Army  use 
conraiercial,  off-lhe-shelf  microcomputers  instead  of  computers  built  to  unique 
DoD  specifications.  We  made  that  recommendation  because  off-the-shelf 
computers,  readily  available  in  the  commercial  marketplace,  met  all  Army 
established  performance  criteria  at  about  one-third  the  cost.  As  discussed  in  this 
report,  similar  circumstances  do  not  exist  for  SPS.  Because  no  existing 
procurement  software  product  or  proposed  combination  of  products  will  meet 
SPS  requirements,  all  software  proposed  will  have  to  be  enhanced  to  meet  SPS 
functional  requirements.  The  amount  of  software  development  ultimately 
required  will  be  directly  related  to  how  well  the  stated  functional  requirements 
fully  reflect  user  needs. 

We  fully  recognize  that  the  SPS  program  is  overseen  by  Agency  and  Office  of 
the  Secretary  of  Defense  review  boards.  We  also  recognize  that  such  oversight 
is  no  guarantee  that  SPS  will  be  acquired  on  schedule  or  at  originally  anticipated 
costs.  The  contracting  methodology  being  used  to  acquire  SPS  is  new  and  no 
"lessons  learned"  exist  to  help  reduce  potential  risks.  Only  time  will  ultimately 
define  the  methodology's  effectiveness  in  meeting  all  DoD  user  requirements  at 
a  fixed  price. 

Since  we  performed  the  audit,  the  Agency  has  changed  the  type  of  contract  it 
intends  to  use  for  SPS.  Instead  of  a  "requirements"  type  of  contract,  it  now 
intends  to  issue  an  indefinite  delivery,  indefinite  quantity  contract,  which  limits 
the  minimum  obligation  to  DoD  to  less  than  $7  million.  We  have  no  doubt  as 
to  whether  the  SPS  acquisition  program  is  worth  a  $7  million  investment. 
However,  we  continue  to  believe,  as  discussed  in  the  report  and  in  this 
appendix,  that  the  contracting  methodology  selected  to  acquire  SPS  is  risky. 

Management  Comments  on  SPS  Functional  Requirements.  The  Director, 
Defense  Procurement,  stated  that  the  requirements  in  the  SPS  solicitation  were 
distilled  from  those  functional  requirements  originally  defined  by  the  user 
community  and  were  reviewed  by  the  Procurement  CIM  Council  (now  the  SPS 
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Council)  to  verify  that  those  requirements  should  be  automated.  Deliberately 
constructed  to  avoid  dictating  a  specific  solution  or  method  of  achieving  DoD 
requirements,  the  solicitation  contained  a  matrix  that  tied  the  solicitation 
requirements  to  the  user-developed  requirements.  That  matrix  was  reviewed  by 
a  functional  community  working  group  to  verify  that  all  user-defined 
requirements  that  should  be  automated  were  in  the  solicitation.  Additionally, 
the  Director,  Defense  Procurement,  stated  that  the  report's  statement  that 
functional  requirements  were  consolidated  to  comply  with  FASA  was  incorrect 
because  the  Federal  requirement  to  express  agency  needs  in  terms  of 
performance  specifications  or  functional  requirements  predates  FASA. 

The  Agency  commented  that  the  original  user  requirements  in  the  SPS 
Functional  Description  were  provided  to  the  SPS  contracting  officer  for 
inclusion  in  the  solicitation,  but  the  Functional  Description  was  determined  to 
be  similar  to  a  "detailed"  specification,  describing  "what"  was  required  and 
"how"  to  provide  the  function.  In  accordance  with  DoD  guidance  and  with  the 
concurrence  of  senior  representatives  of  the  procurement  community,  the 
contracting  officer  rewrote  the  original  user  requirements  to  be  more 
representative  of  a  performance  specification.  The  Agency  also  described  the 
processes  implemented  to  help  ensure  that  the  offerors  folly  understand  the 
requirements  of  SPS.  Through  those  processes,  any  misunderstanding  of  the 
requirements  by  an  offeror  will  be  resolved  and  validated  solutions  will  be 
incorporated  in  the  final  contracts. 

Audit  Response.  We  agree  with  the  process  management  described  as  to  the 
evolution  of  the  solicitation's  requirements.  We  also  agree  that  the 
requirements  as  stated  in  the  solicitation  were  deliberately  constructed  as 
performance  specifications  to  avoid  dictating  a  specific  solution  or  method  of 
achieving  DoD  requirements.  Our  concern  focuses  on  whether  the  performance 
specifications  are  specific  enough  to  adequately  convey  the  foil  needs  of  the  user 
to  the  offerors.  Further,  that  concern  is  heightened  because  software 
development  is  required  to  meet  all  stated  requirements.  Broadly  stated 
requirements  are  not  conducive  to  developing  effective  software.  Although 
processes  have  been  established  to  help  make  sure  potential  contractors  well 
understand  the  requirements  of  SPS,  the  effectiveness  of  those  processes  is  yet 
to  be  proven. 

We  q^uestion  some  Director,  Defense  Procurement,  comments.  We  found  no 
provision  in  the  SPS  Council  charter  for  the  Council  to  determine  which 
functional  requirements  "should  be  automated."  We  believe  the  Council  may 
have  difficulty  in  convincing  users  of  legacy  systems  that  some  automated 
support  presently  provided  by  those  systems  may  not  need  to  be  provided  by 
SPS.  Also,  the  matrix  of  requirements  in  the  solicitation  does  not  directly  tie  to 
the  original  700  user-developed  requirements.  Instead,  it  reflects  the  300 
requirement  statements,  which  were  distilled  from  the  Functional  Description. 
We  agree  that  Federal  and  DoD  guidance  emphasized  the  use  of  commercial 
products  before  FASA.  We  cited  FASA  as  the  impetus  for  using  performance 
specifications  in  the  SPS  solicitation  because  the  Director,  Defense 
Procurement,  in  the  October  1994  memorandum  initiating  the  SPS  procurement 
process,  stated  that  the  procurement  should  be  conducted  in  accordance  with 
FASA. 
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Management  Comments  on  SPS  Software  Development.  The  Director, 
Defense  Procurement,  stated  that  none  of  the  commercial  software  products 
examined  during  market  research  in  July  1994  could  satisfy  all  DoD 
requirements,  llie  Director  inferred,  however,  that  a  combination  of  those 
products  could  meet  all  DoD  requirements  and  that  the  combined  product  would 
be  offered  to  DoD  through  anticipated  teaming  arrangements.  The  Director  also 
stated  that  all  SPS  functional  requirements  do  not  have  to  be  satisfied  initially, 
but  will  be  required  about  2  years  after  contract  award.  The  Director  did  not 
agree  that  the  SPS  solicitation's  inclusion  of  software  process  questionnaires  and 
required  software  capability  evaluations  indicated  that  substantial  software 
development  was  anticipated.  The  questionnaires  were  included  solely  to 
evaluate  an  offeror's  ability  to  manage  the  integration  of  multiple  software 
products  because  teaming  arrangements  were  anticipated.  Similarly,  the 
requirement  for  software  engineering  support  is  to  provide  for  interfacing  with 
other  automated  systems,  a  common  and  customary  commercial  practice. 

The  Defense  Logistics  Agency  stated  that  the  report  failed  to  recognize  DoD 
efforts  to  reduce  the  need  for  software  development  by  the  offerors  and  did  not 
explain  opportunities  available  for  potential  offerors  to  enhance  their  software  to 
provide  substantial  amounts  of  the  required  SPS  functionality.  The  Agency  also 
cited  aspects  of  the  SPS  procurement  strategy  that  help  to  reduce  the  risks  of  a 
fixed-price  contract.  One  risk  mitigation  factor  is  the  "fly-off"  between 
competing  vendors  preceding  the  final  vendor  selection.  Another  factor  cited 
was  the  type  of  contract  to  be  awarded.  Because  an  indefinite  delivery, 
indefinite  quantity  contract  will  be  used,  the  selected  vendor  will  have  distinct 
financial  incentives  to  deliver  the  functionality  promised  and  in  the  manner 
promised. 

Audit  Response.  While  the  Director,  Defense  Procurement,  may  have 
anticipated  SPS  proposals  that  would  immediately  and  fully  satisfy  DoD 
functional  requirements,  such  proposals  were  not  made.  Instead,  the  SPS 
proposals  indicate  that  the  commercial  softwares  proposed  will  have  to  be 
substantially  enhanced  to  provide  the  functionality  required.  We  continue  to 
believe,  as  discussed  in  the  report,  that  the  solicitation  indicated  that  a  potential 
need  for  substantial  software  development  was  recognized  from  the  onset. 
Accordingly,  we  have  serious  doubt  that  SPS  should  be  acquired  as  a 
"commercial  item,"  as  defined  by  FASA.  More  importantly,  a  fixed-price 
contracting  mechanism,  as  required  when  procuring  a  "commercial  item,"  is 
risky  when  substantial  software  development  is  required.  As  previously 
discussed,  that  risk  is  compounded  when  user  requirements  are  not  well-defined. 

As  to  the  Director,  Defense  Procurement,  comments  on  the  purpose  for 
requiring  software  process  questionnaires,  we  believe  an  inappropriate  tool  may 
have  been  chosen  to  assess  an  offeror's  ability  to  manage  the  integration  of 
software  packages.  Addendum  M  of  the  solicitation  clearly  states  that  each 
offeror's  software  process  will  be  evaluated  by  using  the  Software  Capability 
Evaluation  Method,  a  methodology  developed  by  die  Software  Engineering 
Institute.  Those  evaluations  will  establish  the  relative  maturity  of  each  offeror's 
software  development  process.  The  focus  of  the  evaluations  will  be  on  an 
offeror's  ability  to  efficiently  develop  software  of  high  quality,  not  on  the 
ability  to  manage  the  integration  of  software  packages.  Also,  we  agree  that  SPS 
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will  have  to  be  interfaced  with  other  automated  systems  and  that  interfaces  are  a 
common  and  usual  requirement.  The  SPS  solicitation  clearly  states  that 
interface  development  will  be  provided  through  software  engineering  support 
taskings.  However,  software  engineering  support  is  not  limited  to  interface 
development  and  could  be  used  to  develop  software  related  to  other  SPS 
functional  requirements.  Accordingly,  we  believe  that  the  requirement  for 
software  engineering  support  is  an  indicator  that  software  development  is 
anticipated. 

We  agree  with  Agency  comments  that  the  draft  report  did  not  recognize  prior 
efforts  to  reduce  the  need  for  future  software  development.  We  have  revised 
the  report  to  recognize  those  efforts. 

Management  Comments  on  Site  Requirements.  The  Director,  Defense 
Procurement,  stated  that  the  recent  conversion  to  an  indefinite  delivery, 
indefinite  quantity  type  of  contract  permits  orders  to  be  placed  for  a  variety  of 
matrixed  scenarios.  Those  scenarios,  designed  to  accommodate  varied  site 
factors  such  as  number  of  users,  hardware,  and  networks,  will  substantially 
reduce  the  risk  of  fixed  prices  when  the  environments  of  individual  sites  are  not 
well  defined. 

The  Defense  Logistics  Agency  stated  that  defining  existing  site  requirements 
would  waste  Government  and  indus^  time.  Regarding  legacy  system 
functional  requirements,  SPS  is  the  designated  "purple  suit"  solution;  the  DoD 
standard  procurement  system  of  the  future.  Accordingly,  the  procurement 
processes  represented  by  the  legacy  systems  have  little  applicability. 
Additionally,  the  development  of  system  interfaces  will  be  tasked  as  close  to  the 
actual  time  of  need  as  practicable.  Developing  detailed  interface  requirements 
before  that  time  would  be  impractical  because  these  interfaces  are  constantly 
being  changed.  Further,  the  existing  physical  infrastructure  at  procurement  sites 
is  being  upgraded  to  meet  DoD-wide  standards.  Therefore,  developing  detailed 
descriptions  of  the  present  procurement  physical  operating  environments  would 
be  meaningless.  The  SPS  solicitation  describes  the  operating  and  physical 
environment  requirements  for  meeting  DoD  standards. 

Audit  Response.  Based  on  the  changes  made  to  the  SPS  solicitation  and  other 
factors  described  in  management's  comments,  we  no  longer  believe  that  the  lack 
of  information  describing  physical  site  environments  and  architectures  poses  a 
material  hindrance  to  developing  a  sound  and  reasonable  fixed-price  proposal. 
Accordingly,  we  deleted  that  portion  of  the  draft  report  that  discussed  the 
differences  in  site  architectures.  However,  we  are  still  concerned  that  other 
vague,  site-related  SPS  requirements,  such  as  system  interfaces,  may  prove 
extremely  difficult  to  confidently  cost  and  price.  Regarding  the  Agency's 
comments  on  system  interface  requirements,  we  are  not  concerned  about  the 
timing  of  the  tasks  to  develop  those  interfaces.  We  are  concerned  whether  the 
estimate  of  allocated  hours  is  sufficient  to  build  interfaces  to  present  and  future 
systems  about  which  relatively  little  is  defined.  The  Agency  also  suggested  that 
accommodating  the  unique  processes  used  by  differing  sites  would  be  irrelevant 
because  SPS,  as  the  "standard"  DoD  procurement  system,  would  become  the 
single,  DoD-wide  way  of  doing  procurement  business.  We  believe  that  outlook 
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is  optimistic.  If  SPS  does  not  equal  or  exceed  the  functionality  presently 
provided  through  the  various  legacy  systems,  sites  using  those  legacy  systems 
will  have  little  incentive  to  adopt  SPS. 

Management  Comments  on  Alternative  Deployment  Approaches.  The 
Defense  Logistics  Agency  stated  that  although  the  SPS  Program  Manager  has 
stated  an  intended  deployment  approach,  execution  of  that  intent  has  not  been 
formally  requested  of,  or  approved  by,  the  MAISRC.  That  stated  approach  is 
to  initially  field  SPS  to  manual  or  semi-automated  procurement  sites.  Early 
SPS  implementation  at  those  sites  will: 

-  help  reduce  DoD  problems  with  unmatched  disbursements, 

-  exploit  Electronic  Commerce/Electronic  Data  Interchange  capabilities 
as  rapidly  as  possible,  and 

-  obtain  estimated  procurement  benefits  of  $76  million. 

Those  reasons  will  be  presented  at  the  SPS  Milestone  II/IIIA  briefing  to  the 
MAISRC.  The  Program  Manager  believes  the  MAISRC  will  approve  die 
intended  approach  and  achieve  early  SPS  benefit  instead  of  waiting  for 
interfaces  to  be  developed,  a  necessity  for  fielding  SPS  at  procurement 
organizations  using  legacy  systems. 

Audit  Response.  We  agree  that  the  intended  deployment  approach  has 
advantages,  as  acknowledged  in  the  draft  report.  However,  management 
comments  do  not  fully  address  our  stated  concern.  The  SPS  Program  Manager 
has  not  quantitatively  established  the  associated  costs  and  benefits  of  deploying 
SPS  to  legacy  sites  first.  We  believe  that  financial  considerations  should  also  be 
a  factor,  along  with  obtaining  early  benefits,  in  program  planning  and 
execution.  The  financial  analysis  of  alternative  deployment  approaches  should 
not  be  deferred  until  the  Milestone  II/IIIA  review,  now  scheduled  for  mid- 1997. 
Should  the  analysis  provide  financial  reason  to  alter  the  deployment  approach,  it 
may  not  be  practical  to  implement  changes  at  that  time. 

Management  Comments  on  SPS  Shared  Data  Warehouse.  The  Defense 
Logistics  Agency  concurred  that  the  significance  of  the  SPS  Shared  Data 
Warehouse  has  not  been  stressed  within  existing  documentation  or  project  plans. 


Management  Comments  on  Finding  B 


The  Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 
Intelligence)  disagreed  with  draft  Finding  B,  "Standard  Procurement  System 
Testing  Strategy,"  as  written,  but  provided  no  specific  comments  on  the  topics 
discussed.  The  Director,  Defense  Procurement,  did  not  comment  on  the  finding 
or  discussion.  The  Defense  Logistics  Agency  partially  concurred  with  the 
finding  and  commented  on  the  topics  discussed.  A  summary  of  management 
comments  on  Finding  B  and  the  audit  response  follows. 
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Appendix  E.  Summary  of  Management  Comments  and  Audit  Response 


Management  Comments  on  Finding  B.  The  Assistant  Secretary  of  Defense 
(Command,  Control,  Communications,  and  Intelligence),  with  input  from  the 
Director,  Operational  Test  and  Evaluation,  and  the  Deputy  Director,  Test, 
Systems  Engineering  and  Evaluation,  stated  that  the  finding  should  be  reworded 
to  emphasize  the  criticality  of  validated  user  operational  requirements  to  the 
formulation  of  a  testing  strategy  that  reasonably  ensures  user  requirements  will 
be  met.  The  Assistant  Secretary  also  stated  that  the  SPS  Council  had  formed  an 
SPS  Working  Integrated  Product  Team  to  develop  the  SPS  operational 
requirements  document,  upon  which  the  TEMP  for  Increment  One  operational 
test  and  evaluation  will  be  based. 

The  Defense  Logistics  Agency  partially  concurred,  but  cited  on-going  efforts,  as 
described  in  the  following  topical  summaries,  to  improve  planned  SPS  testing. 
Those  improvements  will  reasonably  ensure  that  testing  will  identify  unmet  user 
requirements. 

Management  Comments  on  Operational  Requirements.  The  Defense 
Logistics  Agency  stated  that  the  SPS  Council  formed  a  Working  Integrated 
Product  Team  to  develop  a  SPS  Operational  Requirements  Document.  The 
Operational  Requirements  Document  is  the  primary  source  from  which 
operational  performance  requirements  are  normally  derived.  However,  because 
the  SPS  testers  needed  user- validated  performance  requirements  to  test  against 
and  the  procurement  community  had  not  provided  those  requirements,  the  SPS 
Program  Manager  gathered  operational  performance  requirements,  along  with 
respective  performance  thresholds  and  objectives,  from  nine  representative  SPS 
sites.  The  SPS  Program  Manager  is  now  working  with  Office  of  the  Secretary 
of  Defense  test  organizations,  members  of  the  user  community,  and 
developmental  and  operational  test  evaluators  in  iterative  integrated  product 
team  meetings  to  facilitate  the  development  of  a  user-validated  Operational 
Requirements  Document  and  derivative  operational  test  requirements.  The 
validated  Operational  Requirements  Document  will  be  provided  no  later  than  60 
days  prior  to  the  Operational  Test  Readiness  Review  and  the  validated 
operational  performance  requirements  will  be  incorporated  into  the  TEMP  and 
the  SPS  Acquisition  Program  Baseline. 

The  Agency  further  stated  that  the  functionality  provided  in  each  increment  will 
not  be  defined  until  contract  award.  Contract  award  will  be  partially  based  on 
the  amount  of  total  SPS  functionality  currently  available  or  presently  being 
developed  in  the  proposed  commercial  product.  Also,  negotiations  will  precede 
contract  award  to  help  determine  the  fonctionality  to  be  provided  in  each  SPS 
increment.  Accordingly,  the  SPS  Council  does  not  need  to  define  the 
functionality  of  each  increment  for  testing  purposes. 

Audit  Response.  The  SPS  Program  Management  Office,  in  collaboration  with 
the  varied  members  of  the  testing  community,  has  made  commendable  progress 
in  strengthening  testing  plans  and  establishing  test  criteria  since  we  completed 
the  audit.  We  note  die  SPS  program  management  office  has  embraced  the 
integrated  product  team  concept  and  convincingly  demonstrated  the  advantages 
offered  by  that  concept. 
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Appendix  £.  Summary  of  Management  Comments  and  Audit  Response 


Management  Comments  on  Compressed  Schedule.  The  Defense  Logistics 
Agency  did  not  agree  that  the  SPS  testing  schedule  was  compressed  or  that  the 
schedule  has  insufficient  test  planning  time  or  test  execution  time.  Rather,  the 
Agency  feels  that  the  SPS  testing  schedule  is  better  described  as  being 
aggressive.  Additionally,  the  Office  of  the  Secretary  of  Defense  testing 
community  is  now  participating  in  all  SPS  test  planning  through  the  integrated 
product  team  approach  and  has  voiced  no  dissatisfaction  with  the  SPS  testing 
approach  or  time  lines. 

Audit  Response.  If  the  testing  community  is  genuinely  satisfied  that  the  SPS 
testing  schedule  provides  for  sufficient  test  planning  and  test  execution  time, 
then  we  agree  that  compliance  with  related  DoD  policy  requirements  is  of  less 
importance.  It  would  be  advisable  for  the  testing  community  to  endorse  the  test 
schedule  explicitly,  because  silence  is  ambiguous. 

Management  Comments  on  Testing  for  Capability  to  Share  Data.  The 
Defense  Logistics  Agency  concurred  that  the  SPS  Shared  Data  Warehouse  test 
strategy  needs  to  be  developed  and  integrated  into  the  overall  SPS  test  strategy. 


Other  Management  Comments 


Management  Comments  on  Other  Matters  of  Interest.  The  Director, 
Defense  Procurement,  stated  that  the  report  was  misleading;  the  responsibilities 
and  duties  of  the  SPS  Council  is  considerably  broader  than  providing  user 
requirements  and  perspective  to  the  SPS  program  manager.  The  Council's 
charter  assigns  the  Council  the  responsibility  for  "planning,  developing, 
coordinating,  and  recommending  improved  business  practices."  The  Council 
also  monitored  the  execution  of  Procurement  Functional  Steering  Committee 
decisions  and  managed  the  Procurement  CIM  Functional  Integration 
Management  Organization.  The  Director  stated  that  it  is  not  a  requirements 
generating  body,  but  is  responsible  under  its  charter  for  assuring  that 
requirements  that  should  be  automated  are  automated. 

Audit  Response.  We  agree  with  the  Director's  comments  that  the  Council  had 
extensive  responsibilities.  Additionally,  we  agree  that  the  Council  is  not  a 
requirements  generating  body;  requirements  come  from  the  procurement 
community.  However,  upon  initiation  of  the  SPS  acquisition,  the  primary  role 
of  the  Council  has  been  to  represent  the  user  and  provide  user  requirements  and 
other  considerations  of  the  user  to  the  SPS  Program  Management  Office.  As 
previously  stated,  we  found  no  provision  in  the  charter  that  tasks  the  Council  to 
determine  which  requirements  should  be  automated  or  to  assure  that  those 
requirements  are  automated. 
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Appendix  F.  Organizations  Visited  or  Contacted 


Office  of  the  Secretary  of  Defense 

Under  Secretary  for  Acquisition  and  Technology 

Deputy  Under  Secretary  of  Defense  (Acquisition  Reform),  Washington,  DC 
Director,  Defense  Procurement,  Washington,  DC 
Director,  Operational  Test  and  Evaluation,  Washington,  DC 
Director,  Test  Systems  Engineering  and  Evaluation,  Washington,  DC 
Under  Secretary  of  Defense  (Comptroller) 

Director,  Program  Analysis  and  Evaluation,  Arlington,  VA 
Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and  Intelligence) 
Deputy  Assistant  Secretary  for  Acquisition,  Washington,  DC 


Department  of  the  Army 

Assistant  Secretary  for  Research,  Development,  and  Acquisition 

Deputy  Assistant  Secretary  for  Financial  Operations,  Washington,  DC 
Deputy  Assistant  Secretary  for  Procurement,  Washington,  DC 


Department  of  the  Navy 

Assistant  Secretary  for  Research,  Development,  and  Acquisition 

Deputy  Assistant  Secretary  for  Acquisition  and  Business  Management, 
Arlington,  VA 

Naval  Iirformation  Systems  Management  Center,  Arlington,  VA 
Naval  Supply  Systems  Command 

Deputy  Commander  for  Contracting  Management,  Arlington,  VA 


Department  of  the  Air  Force 

Assistant  Secretary  for  Acquisition 

Deputy  Assistant  Secretary  for  Contracting,  Washington,  DC 


Other  Defense  Organizations 

Defense  Information  Systems  Agency 
Joint  Interoperability  Test  Center 

Developmental  Test  Organization,  Washington,  DC 
Operational  Test  Organization,  Washington,  DC 
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Appendix  F.  Organizations  Visited  or  Contacted 


Other  Defense  Organizations  (Cont’d) 

Defense  Logistics  Agency 

Defense  Contract  Management  Command 

Director  for  Contract  Management,  Fort  Belvoir,  VA 
Director,  Defense  Procurement  Corporate  Information  Management  Systems 
Center,  Fort  Belvoir,  VA 
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Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition  and  Technology 
Director,  Defense  Procurement 

Director,  Defense  Logistics  Studies  Information  Exchange 
Director,  Operational  Test  and  Evaluation 
Director,  Test  Systems  Evaluation  and  Engineering 
Under  Secretary  of  Defense  (Comptroller) 

Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 

Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and  Intelligence) 


Department  of  the  Army 

Assistant  Secretary  of  the  Army  (Financial  Management  and  Comptroller) 
Assistant  Secretary  of  the  Army  Research,  Development,  and  Acquisition) 
Auditor  General,  Department  of  the  Army 


Department  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Financial  Management  and  Comptroller) 
Assistant  Secretary  of  the  Navy  (Research,  Development,  and  Acquisition) 
Auditor  General,  Department  of  the  Navy 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Assistant  Secretary  of  the  Air  Force  (Acquisition) 

Auditor  General,  Department  of  the  Air  Force 


Other  Defense  Organizations 

Director,  Defense  Contract  Audit  Agency 
Director,  Defense  Finance  and  Accounting  Service 

Director,  Defense  Finance  and  Accounting  Service  Indianapolis  Center 
Director,  Defense  Information  Systems  Agency 
Director,  Defense  Logistics  Agency 

Commander,  Defense  Contract  Management  Command 

Director,  Defense  Procurement  Corporate  Information  Management  Systems  Center 
Director,  National  Security  Agency 

Inspector  General,  National  Security  Agency 
Inspector  General,  Defense  Intelligence  Agency 
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Non-Defense  Federal  Organizations  and  Individuals 

Office  of  Management  and  Budget 

Technical  Information  Center,  National  Security  and  International  Affairs  Division, 
General  Accounting  Office 

Chairman  and  ranking  minority  member  of  each  of  the  following  congressional 
committees  and  subcommittees: 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 

Senate  Subcommittee  on  Acquisition  and  Technology,  Committee  on  Armed 
Services 

Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  National  Security,  Committee  on  Appropriations 
House  Committee  on  Government  Reform  and  Oversight 
House  Subcommittee  on  National  Security,  International  Affairs,  and  Criminal 
Justice,  Committee  on  Government  Reform  and  Oversight 
House  Committee  on  National  Security 
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Part  in  -  Management  Comments 


Assistant  Secretary  of  Defense  (Command, 
Control,  Communications,  and  Intelligence) 
Comments 


OFFICE  OF  THE  ASSISTANT  SECRETARY  OF  DEFENSE 
6000  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-6000 


Bis  JUL  SK 


COMMAND.  CONTROL. 
COMMUNICATIONS.  AND 
INTELUGENCE 


MEMORANDUM  FOR  DIRECTOR,  READINESS  AND  OPERATIONAL  SUPPORT 
DIRECTORATE,  DODIG 

SUBJECT;  Audit  Report  on  Allegations  to  the  Defense  Hotline 
Concerning  the  Standard  Procurement  System 
(Project  No.  5RE-8019) 


We  appreciate  the  opportunity  to  comment  on  this  draft  audit 
report.  Our  comments  have  incorporated  inputs  from  tne  office  of 
the  Director,  Operational  Test  and  Evaluation  and  the  office  of 
the  Deputy  Director,  Test,  Systems  Engineering  &  Evaluation, ^ 
OUSD(AScT) .  We  generally  agree  that  major  automated  information 
system  (AIS)  programs,  such  as  the  Standard  procurement  System 
(SPS)  program,  have  acquisition  risks.  Without  approved  user 
operational  requirements,  program  testing  strategies  may  not  be 
appropriate  for  reducing  the  risk  that  user  needs  may  not  be  met. 

However,  Major  Automated  Information  System  Review  Council 
(MAISRC)  staff  oversight  reviews  focus  on  how  major  AIS  programs 
are  mitigating  acquisition  and  program  risks .  Because  MAISRC 
staff  members  have  participated  in  SPS  working-level  integrated 
product  teams,  we  believe  the  SPS  Program  Manager  has 
continues  to  establish  appropriate  processes  for  minimizing  SPS 
program  risks. 

As  requested,  the  attached  comments  address  the  conclusions 
and  recommendations  that  involve  the  MAISRC.  If  you  have 
questions  regarding  the  attached  comments,  please  contact  my 
action  officer,  David  Dore,  at  (703)  681-4989. 


Deputy  Assist^t  Secretary  of  Defense 
(C3I  Acquisition) 

Attachment 


o 
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Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 

Intelligence)  Comments 


CQMMBWTS  QN 

DRAFT  AUDIT  REPORT  CONCERNING  THE 
STANDARD  PROCUREMENT  SYSTEM  (5RE-8019) 


The  draft  report  states  the  DODIG  concern  that  the  strategy 
for  acquiring  SPS  adds  considerable  program  risk  (Finding  A)  and 
there  is  a  need  for  DOD,  before  SPS  contract  award,  to  reexamine 
the  SPS  acquisition  and  program  strategies  to  determine  if  the 
inherent  risks  are  still  acceptable.  Reviews  of  acquisition  and 
program  risks  are  essential  parts  of  the  oversight  process .  In 
fact  the  MAI  SRC  Milestone  I  decision  memorandum  directed  the 
MAISRC  be  provided  the  SPS  strategies  for  dealing  with  SPS 
technical  risks.  With  MAISRC  staff  participation  and  support, 
the  SPS  migration  strategies  for  managing  program  and  technical 
risk  were  completed  on  November  15,  1995,  and  implementing 
actions  continue.  However,  because  of  the  DODIG  concern,  a 
MAISRC  WIPT  will  again  reexamine  these  strategies  within  90  days 
of  contract  award. 

The  other  DODIG  finding  states  that  inadequate  testing 
strategies  increased  the  risk  that  SPS  may  not  meet  user 
requirements.  We  do  not  agree  that  this  is  the  case  with  the 
SPS,  based  on  the  information  in  the  draft  audit  report. 

Instead,  we  agree  that  the  lack  of  validated  user  operational 
requirements  may  result  in  an  SPS  testing  strategy  which 
increases  the  risk  that  the  program  may  not  meet  user 
requirements.  The  SPS  Council  has  established  an  SPS  WIPT  to 
develop  the  operational  requirements  document  (ORD) .  Th'is  ORD 
will  be  approved  prior  to  approval  of  the  final  TEMP  supporting 
formal  increment  1  operational  test  and  evaluation. 
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Director,  Defense  Procurement,  Comments 


OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 
WASHINGTON,  DC  20301 

July  15,  1996 

ACQUISITION 


MEMORANDUM  FOR  INSPECTOR  GENERAL,  DEPARTMENT  OF  DEFENSE 


SUBJECT:  Draft  proposed  audit  report  on  allegations  to  the 

Defense  Hotline  concerning  the  Standard  Procurement 
System  (Project  No.  5RE-8019) . 

A  response  to  recommendation  B.l.  of  the  draft  proposed 
report  is  attached.  Please  note  that  the  Standard  Procurement 
System  Request  for  Proposals  was  amended  subsequent  to 
performance  of  the  Audit  and,  consequently,  the  report's  findings 
and  recommendations  should  be  re-evaluated. 

I  have  also  attached  some  additional  comments  for  your 
cons idera t ion . 


Eleanor  R.  Spector 
Director,  Defense  Procurement 


Attachments 


Director,  Defense  Procurement,  Comments 


DDP  Comments 

DRAFT  PROPOSED  AUDIT  REPORT  dated  May  15,  1996 
Project  No.  5RE-8019 

Allegations  to  the  Defense  Hotline  Concerning 
the  Standard  Procurement  System 


Recommendationg  for  Corrective  Action 


B  t  I  fAf- 

Identify  and  validate  the  operational  retjuirements  for  the 
Standard  Procurement  System. 

DPP  RegpOhS.S 

Recommendation  accomplished.  Solicitation  amendment  No.  7 
(April  22,  1996)  re-states  and  more  clearly  identifies  the 
operational  environments  in  which  the  SPS  software  application 
must  operate.  The  functions  the  software  must  perform  in  those 
environments  are  identified  in  the  solicitation.  The  functions 
and  environments  were  reviewed  and  validated  prior  to  the 
amendment ' s  release . 


B.l.b. 

Provide  the  operational  performance  requirements  to  the 
Director/  Defense  Procurement  Corporate  Information  Management 
Systems  Center,  Defense  Logistics  Agency,  for  inclusion  in  the 
Standard  Procurement  System  Integrated  Test  Program  Schedule. 

ppp  Response 

Concur.  An  Operational  Requirements  Document  (ORD)  is  being 
developed  and  will  be  available  for  OPEVAL. 


B. l.C. 


Define  the  requirements  for  the  shared  data  base  portion  of 
the  Standard  Procurement  System  and  provide  them  to  the  program 
manager  for  incorporation  into  the  Test  and  Evaluation  Master 
Plan. 


♦ 


Final  Report 
Reference 


Revised 
Page  23 


Revised 
Page  23 
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Director,  Defense  Procurement,  Comments 


Final  Report 
Reference 


Page  4 


DPP  Response 

Partially  concur.  The  SPS  Program  Manager  has  been  advised 
that  the  data  warehouse  must;  store  and  accurately  retrieve  on 
demand  data  needed  by  the  procurement  community;  interface  with 
other  DoD  functional  area  systems  providing  or  requiring 
procurement  related  data;  and,  conform  to  the  DoD  procurement 
data  standards.  More  specific  direction  cannot  be  provided 
currently  because  the  Defense  Data  Repository  System  and 
procurement  data  modeling  activities  are  iterative,  continuous 
processes;  the  types  of  data  stored  will  change  as  user  needs, 
law,  or  regulation  change;  and,  a  firm  identification  of  the 
types  of  data  the  software  will  be  capable  of  inserting  into  the 
warehouse  cannot  be  identified  at  this  time  because  the 
solicitation  contemplates  the  award  of  tailored  contracts  with 
tailored  deliverables. 


Other  Comments 


1 .  Pages  3  and  4 

The  acquisition  approach  discussion  should  be  modified  to 
reflect  the  contractual  arrangement  contemplated  by  the  current 
Request  for  Proposals  (RFP) . 

2  .  Page  6,  Finding  ^Strategy  for  Acguirina  the  SPS^ 

The  discussion  alleges  SPS  functional  requirements  are  too 
broad  and  will  increase  the  likelihood  that  the  SPS  will  not  meet 
specific  user  needs. 

The  functional  requirements  contained  in  the  solicitation 
were  distilled  from  the  user  community  developed  fi^nctional 
requirements  and  reviewed  by  the  SPS  Council  to  verify  that 
requirements  that  should  be  automated  would  be  included  in  the 
solicitation.  The  solicitation  was  deliberately  constructed  to 
avoid  dictating  a  solution  or  method  for  achieving  DoD's 
requirements.  In  effect,  the  offerors  were  provided  performance 
requirements  rather  than  detail  specifications.  To  assure  that 
all  user  requirements  were  addressed  in  the  solicitation,  the 
solicitation  contained  a  matrix  that  tied  the  solicitation 
requirements  back  to  the  user  developed  requirements.  A 
functional  community  working  group  reviewed  the  matrix  and 
verified  that  all  user  defined  requirements  that  should  be 
automated  were  addressed  in  the  solicitation. 
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Director,  Defense  Procurement,  Comments 


a.  The  first  subparagraph  states  "No  known  commercial 
software  product  can  meet  total  SPS  requirements  at  contract 
award . " 

Market  research  conducted  in  July  1994  suggested  that  while 
none  of  the  products  examined  at  that  time  could  satisfy  all  DoD 
requirements,  combinations  of  products  could  satisfy  those 
requirements.  Teaming  arrangements  were  anticipated  and  some  of 
the  offerors  have  formed  teams.  The  offerors  do  not  have  to 
satisfy  the  full  SPS  requirement  at  the  time  of  initial  contract 
award.  The  SPS  delivery  and  deployment  requirements  are 
incremental  processes  that  will  be  tailored  to  each  offeror's 
product.  The  initial  software  release  will  provide  at  least  the 
functionality  contained  in  the  offeror's  current  off  the  shelf 
product  and  will  serve  as  the  testing  baseline.  Each  successive 
release  will  provide  additional  functionality  or  commercial 
enhancements.  Full  SPS  functionality  is  required  approximately 
two  years  following  initial  contract  award. 

b.  The  second  subparagraph  suggests  that  the  software 
process  questionnaires  are  not  a  typical  commercial  practice  and 
indicate  that  substantial  software  development  is  anticipated. 

It  was  anticipated  that  offerors  would  team  or  otherwise 
enhance  existing  software  to  satisfy  the  SPS  requirement. 

Teaming  might  involve  integrating  two  or  more  software  packages. 
Therefore,  it  was  considered  prudent  to  obtain  confidence  in  an 
offeror's  software  management  processes.  The  questionnaires  are 
intended  solely  for  that  purpose.  They  are  not  an  indication 
that  the  successful  offeror  will  have  to  perform  substantial 
software  development , 

4 .  Page  7,  Software  Engineering  Support 

The  SPS  software  application  must  interface  with  existing 
procurement  legacy  systems  and  automated  systems  that  will  be 
used  by  other  functional  communities.  Several  offerors  have 
confirmed  that  commercial  customers  often  ask  to  have  a  vendor's 
commercial  product  interface  with  the  customer's  existing 
systems.  The  development  of  those  interfaces  is  not  considered 
high  risk  effort  and  is  a  customary  commercial  practice. 
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Pages  7  and  8 


Page  9 


5 .  Page  8>  Functional  Regairements 

The  allegation  that  functional  requirements  were 
consolidated  to  comply  with  the  Federal  Acquisition  Streamlining 
Act  of  1994  is  incorrect. 

The  statutory  requirement  to  express  agency  needs  in  terms 
of  performance  specifications  or  functional  requirements  {form, 
fit,  or  function)  predates  the  Federal  Acquisition  Streamlining 
Act  and  the  Federal  Action  Reform  Act.  The  '‘Preference  for  non- 
developmental  items*'  statute,  10  U.S.C.  2325  {repealed  Oct  94), 
required  DoD  to  state  its  requirements  in  terms  of  performance 
desired  or  functions  to  be  performed.  Similar  authority  is 
contained  in  10  U.S.C. 2305. 

See  comment  3.  For  additional  information. 

6.  Pace  10,  Diversity  of  User  Requirementfli 

This  paragraph  suggests  that  the  SPS  solicitation  does  not 
contained  well-defined  site  requirements. 

The  number  of,  and  operating  environment  at,  SPS  sites  is 
fluid  {downsizing  and  consolidation)  and  is  subject  to  change. 

The  current  solicitation  recognizes  that  situation  by  providing  a 
pricing  matrix  which  describes  typical  scenarios  (expressed  in 
terms  of  users,  hardware,  networks,  etc.)  that  might  be 
encountered  at  deployment  sites.  The  contract  type  has  been 
converted  to  an  IDIQ  contract  which  permits  orders  to  be  placed 
for  the  matrixed  scenarios .  If  actual  conditions  at  a  particular 
site  vary  from  the  matrixed  conditions,  an  adjusted  price  would 
be  negotiated.  This  procedure  will  substantially  reduce 
contractor  and  Government  risk. 

7 .  Anoendix  C,,  Other  Matters  of  Interests  SPS  Advisory 

Councils 


The  statement  that  “the  primary  purpose  of  the  Procurement 
CIM  Council  is  to  provide  user  requirements  and  perspective  to 
the  SPS  program  manager*  is  misleading. 

The  Procurement  CIM  Council  {now  titled  the  SPS  Council) 
charter  assigns  the  council  general  responsibility  for  “planning, 
developing,  coordinating,  and  recommending  improved  business 
practices . *  The  council  was  also  directed  to  monitor  the 
execution  of  Procurement  Functional  Steering  Committee  decisions 
and  was  directed  to  manage  a  Procurement  CIM  Functional 
Integration  Management  Organization  (disestablished  upon 
formation  of  the  SPS  Program  Management  Office)  .  It  is  not  a 
requirements  generating  body  but  is  responsible  under  its  Charter 
for  assuring  that  requirements  which  should  be  automated  are 
automated . 
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DEFENSE  LOGISTICS  AGENCY 
HEADQUARTERS 

8725  JOHN  J.  KINGMAN  ROAD.  SUITE  2533 
FT.  BELVOIR,  VIRGINIA  22060-6221 


JUL  1  7 


MEMORANDUM  FOR  THE  ASSISTANT  INSPECTOR  GENERAL  FOR  AUDITING, 
DEPARTMENT  OF  DEFENSE 

SUBJECT:  Draft  Report  on  Allegations  to  the  Defense  Hotline  Concerning  the  Standard 
Procurement  System,  5RE-8019 


IM  REPLY 
REFER  TO 

DDAl 


Enclosed  is  our  response  to  your  request  of  15  May  1 996.  If  you  have  any  questions  please 
call  Dave  Stumpf,  767-6266. 

JACQUELINE  G.  BRYANT 
^  Chief,  Internal  Review  Office 

End 


Ftdtril  fWcycfcng  Progwm  ■  jA  Printed  oo  Rtcycltd  Pip#f 
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AUDIT  TITLE:  Allegations  to  the  Defense  Hotline  Concerning  the 
Standard  Procurement  System,  5RE-8019 

FIHDING  A:  Strategy  for  Acquiring  the  SPS .  The  strategy  for 
acquiring  the  SPS  adds  considerable  risk  to  the  $4.1  billion  SPS 
program.  The  fixed -price  contracting  methodology  used  for  commercial 
items  is  risky  because  SPS  functional  requirements  in  the 
solicitation  are  too  broad  and  because  existing  commercial  software 
requires  substantial  software  development  to  achieve  full  SPS 
functional  capability.  Also,  the  SPS  solicitation  does  not 
sufficiently  define  site  requirements.  Furthermore,  the  program 
manager  did  not  quantitatively  analyze  alternative  deployment 
approaches  or  stress  the  significance  of  the  Share  Data  Warehouse  in 
program  plans  to  assure  the  ultimate  success  of  the  SPS  program. 

As  a  result,  the  needs  of  SPS  users  may  not  be  met  and  actual  costs 
could  exceed  proposed  costs  because  vendors  will  find  it  difficult  to 
provide  realistic  and  comprehensive  cost  proposals  without  well- 
defined  functional  and  site  requirements. 


DLA  COMMENTS:  DLA  partially  concurs  with  the  finding.  DLA  nonconcurs 
with  the  following: 

The  strategy  for  acquiring  SPS  is  commercial  acquisition  of  software 
products  and  related  services  that  when  deployed  on  open  systems 
platforms  and  integrated  into  a  DoD  Common  Operating  Environment, 
meet  the  full  functional,  technical,  performance,  security,  and 
interoperability  needs  of  the  DoD  Procurement  Community.  An 
acquisition  strategy  supporting  commercial  acquisition  over 
government /DoD  unique  in-house  development  is  in  compliance  with  and 
supportive  of  National  Performance  Review,  Defense  Performance 
Review,  and  DoD  objectives  and  directives  for  supporting  dual  use 
technologies,  right-sizing  of  government,  use  of  commercial  off-the- 
shelf  products,  increasing  government  access  to  commercial  state-of- 
the-art  technology,  and  DoD  5000  series  documents.  DIxA  cannot 
understand  nor  support  a  position  which  states  that  a  commercial 
acquisition  strategy  adds  risk  to  DoD  programs,  and  in  particular, 
adds  risk  to  the  SPS  program. 

The  statement  that  SPS  is  a  $4 . IB  program  is  erroneous,  implies  this 
funding  is  under  the  SPS  PM's  control,  and  is  cited  merely  to 
heighten  senior  management  concern  over  potential  risk  to  such  a 
large  dollar  /alue.  SPS  is  defined  and  approved  as  a  10  year 
program.  Tne  estimated  10  year  program  cost  (FYs  95-05)  of  SPS  is 
$327. 9M.  TDtal  SPS  funding  (including  maintenance  and  refresher  user 
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training)  potentially  under  the  control  of  the  PM  during  this  10  year 
period  is  estimated  at  $43 9M.  The  estimated  10  year  Life  Cycle  Cost 
of  SPS  is  $2.9B,  which  contains  substantial  operations  and  support 
costs  contained  within  DoD  Component  budgets  and  outside  the  control 
of  the  SPS  PM.  The  finding  erroneously  uses  a  15  year  cost  estimate 
for  SPS  of  $4. IB,  which  equates  to  full  SPS  operational  capability 
(five  years  to  attain)  plus  10  years  of  operations  and  support  costs. 
As  such,  the  finding  indicates  costs  which  are  five  years  beyond  the 
life  of  the  approved  OSD  program.  The  finding  also  fails  to  note  the 
potential  cost  risk  to  DoD  by  stopping  or  delaying  the  SPS  program 
and  being  forced  to  live  with  the  status  quo.  The  FY  95-05  estimated 
cost  of  the  status  quo  is  $4.4B  and  the  FY  95-10  estimated  cost  of 
the  status  quo  is  $8, IB. 

The  contracting  method  being  used  under  the  SPS  commercial 
acquisition  strategy  is  fair  and  open  competition  for  commercial 
items.  Since  SPS  is  purchasing  a  commercially  available  system, 
which  may  be  modified  or  integrated  with  other  commercial 
applications  to  satisfy  full  SPS  requirements  and  is  based  upon  a 
negotiated  product  delivery  schedule,  the  commercial  items 
requirements  of  FAS  A  94  and  FAR  Part  12  apply  and  require  the  use  of 
fixed  prices. 

The  finding  states  fixed  pricing  is  risky  because  the  SPS 
requirements  are  "too  broad."  The  SPS  requirements  were  taken 
directly  from  the  SPS  Functional  Description,  in  as  detailed  a  form 
as  they  existed,  and  submitted  to  the  SPS  Contracting  Office.  The 
SPS  Contracting  Office  with  the  concurrence  of  the  SPS  Source 
Selection  Advisory  Council  (SSAC)  determined  that  the  requirements 
stated  in  the  SPS  Functional  Description  were  similar  to  a  "detailed" 
specification,  not  only  describing  "what"  was  required,  but  quite 
often,  in  a  duplicative  and  verbose  manner,  describing  "how"  to 
provide  the  functions.  The  SPS  Contracting  Office  with  the 
assistance  of  the  SPS  SSEB  and  other  procurement  functional  users, 
used  the  functions  from  the  Functional  Description  to  create  a 
statement  of  work  more  in  line  with  a  "performance  specification",  as 
required  in  DEPSECDEF  memorandum  of  29  Jun  1994,  subject: 
Specifications  &  Standards  -  A  New  Way  of  Doing  Business.  In  this 
re-write,  every  effort  was  made  to  remove  duplications,  wordiness, 
functions  which  could  be  provided  via  bundled  or  government  provided 
COTS  (e.g.,  RDBMs,  word  processors,  spreadsheets,  etc.),  and 
statements  which  indicated  the  method  in  which  a  function  was  to  be 
provided  instead  of  the  outcome  of  the  function.  The  functional 
approval  body  which  approved  the  SPS  Functional  Description  also 
approved  the  more  performance  related  statement  of  work  which  is 
currently  in  the  SPS  RFP.  Although  the  finding  states  the 
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requirements  are  too  broad,  the  DoD  designated  Senior  Procurement 
representatives  of  the  user  community  approved  this  re-write  as  a 
performance  representation  of  the  original  requirements  and  DLA 
concurs  with  their  position. 


It  should  also  be  noted  that  the  SPS  SSEB  and  contract  negotiations 
team  are  required  to  and  are  in  the  process  of  evaluating  each 
offerors'  proposal.  Clarification  and  deficiency  reports  are  issued 
on  each  rec[uirement  wherein  the  offerors  appear  not  to  understand  or 
perform  a  requirement  in  their  existing  product  or  in  their 
explanation  of  their  approach  toward  providing  a  requirement  within  a 
designated  software  release.  In  addition,  each  offeror's  proposal  is 
subjected  to  a  line-by-line  review,  face-to-face  with  the  offeror  to 
ensure  that  the  requirements  are  understood  and  the  proposed  solution 
meets  the  requirements.  No  contract  can  be  awarded  to  an  offeror 
with  an  outstanding  deficiency.  Strengths /weaknesses  of  approaches 
provided  to  satisfying  requirements  will  be  part  of  the  final 
evaluation/  selection  process.  Through  these  processes,  any 
misunderstandings  on  the  requirements  are  resolved  and  government 
validated  solutions  will  be  part  of  the  final  contracts. 

The  finding  indicates  that  the  fixed  priced  nature  of  the  SPS 
contracting  process  is  risky  because  "substantial  software 
development"  is  required.  The  report  and  finding  fails  to  indicate 
that  prior  to  release  of  the  final  RFP,  a  review  was  conducted  of  the 
requirements  and  a  number  of  requirements  were  removed  where  it  was 
clear  that  substantial  development  was  required.  The  report  and 
finding  failed  to  note  that  although  no  single  commercial  product 
reviewed  a  year  before  the  RFP  was  released  provided  all  required 
functionality,  that  industry  continued  to  improve  their  commercial 
products  during  this  period  and  the  RFP  did  not  preclude  the 
commercial  integration  of  multiple  commercial  products,  COTs,  and 
GOTs  products  which  could  provide  substantial  amounts  of  the  required 
functionality.  However,  some  software  development  is  in  fact 
indicated  in  each  proposed  solution. 

As  stated  above,  every  effort  is  being  taken  to  ensure  each  offeror 
understands  the  requirements,  the  government  is  negotiating  the 
release  schedule  of  functionality,  and  the  offerors'  approach  to 
providing  the  functionality  is  being  evaluated,  as  is  each  offerors' 
ability  to  develop  and  maintain  software.  In  addition,  a  "fly-off" 
between  vendors  will  be  used  during  which  time  the  first  increment  of 
additional  functionality  promised  will  be  delivered  and  evaluated 
before  a  final  downselect  to  a  single  vendor  will  be  made.  These 
efforts  will  greatly  assist  in  fixed  price  risk  mitigation  for 
software  development.  The  most  effective  risk  mitigation  technique 
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remains,  however,  the  power  of  the  purse.  The  SPS  contract  is  an 
IDIQ  contract.  Following  downselect,  the  winning  vendor  will  only 
get  future  delivery  orders  for  the  software  and  upgrades  that  are 
accepted  and  meet  the  government's  functional  recjuirements .  Since 
the  government  is  buying  licenses  for  software  and  upgrades,  for  the 
winning  SPS  vendor  to  succeed  (i.e.,  make  sales),  he  must  provide 
that  functionality  promised,  in  the  manner  promised,  or  no  future 
delivery  orders  will  be  issued. 


The  finding  states  that  the  SPS  solicitation  does  not  define  site 
requirements.  As  was  repeatedly  discussed  during  the  audit,  stating 
the  existing  procurement  site  requirements  in  detail  would  be  a  waste 
of  time  for  the  government  and  industry  from  several  perspectives , 

If  the  audit  is  referring  to  the  legacy  system  functional 
requirements,  the  SPS  represents  a  negotiated  and  Component  Senior 
procurement  official  approved  set  of  standard  procurement  processes 
and  procedures;  the  approved  way  of  doing  procurement  business  in  the 
future,  DoD“Wide.  As  such,  the  SPS  requirement  is  to  satisfy  the 
standard  way  of  doing  procurement  business,  and  the  legacy  system  way 
of  doing  procurement  business  has  little  applicability.  If  the  audit 
is  referring  to  system  interfaces,  these  interfaces  are  undergoing 
continual  change  and  it  would  be  inappropriate  to  state  these 
requirements  in  detail  now,  only  to  have  to  change  them  later.  As 
such,  each  interface  will  be  tasked  out  for  development  to  the 
winning  SPS  vendor  under  an  interface  control  document  and  as  close 
to  the  actual  time  of  need  as  practicable.  If  the  audit  is  referring 
to  the  individual  sites  physical  makeup  (e.g.,  PCS,  LANs,  servers, 
etc.),  the  existing  physical  environment  at  the  DoD  procurement  sites 
is  undergoing  infrastructure  upgrade  to  bring  them  into  DoD 
compliance  and  in  preparation  for  SPS.  As  such,  the  existing 
physical  environment  is  meaningless.  The  SPS  solicitation  does 
describe  the  sites  planned  future  environmental  operating  environment 
and  physical  requirements. 

The  finding  states  that  the  SPS  PM  did  not  "quantitatively  analyze 
alternative  deployment  approaches."  As  was  often  discussed  during 
the  audit,  this  is  merely  a  matter  of  timing  and  what  is  required  at 
this  point  in  the  SPS  life  cycle  management  approval  process.  No 
deployment  approval  decision  for  SPS  has  yet  been  requested.  The  SPS 
PM  did  state  that  his  intention  was  to  deploy  to  manual/semi- 
automated  sites  first,  but  this  intent  has  not  been  requested 
formally  nor  approved.  The  basis  of  this  intention  is  three-fold. 
First,  the  manual /semi -automated  sites  are  large  contributors  to  the 
DoD  unmatched  disbursements  issue,  so  early  implementation  of  SPS  at 
such  sites  helps  reduce  this  major  DoD  problem  area  (i.e., 

$10B/month) .  Second,  by  putting  SPS  at  manual /semi -automated  sites. 


Defense  Logistics  Agency  Comments 


EC/EDI  capabilities  are  provided,  which  furthers  another  DoD  goal  of 
exploiting  this  capability  as  rapidly  as  possible  within  DoD.  Third, 
the  audit  Team  was  provided  the  initial  analysis  of  the  cost  benefits 
from  deploying  to  manual/semi -automated  sites  versus  awaiting  the 
initial  development  of  interfaces  required  to  deploy  at  any  existing 
legacy  procurement  site.  Procurement  benefits  of  $76M  have  been 
estimated  from  this  action  and  is  considered  quantitative  analysis. 

As  such,  the  SPS  PM  sees  the  decision  to  deploy  to  manual/semi - 
automated  sites  as  a  binary  decision  which  the  MAISRC  will  make: 
afford  DoD  early  benefits  by  deploying  to  manual /semi -automated  sites 
or  await  the  development  of  interfaces  before  any  deployment.  All  of 
these  reasons  will  be  presented  to  the  MAISRC  Milestone  Decision 
Authority  when  requesting  deployment  approval  for  SPS  at  the  SPS 
Milestone  II/IIIA  decision  briefing. 

DLA  nonconcurs  with  the  statement  that  the  “needs  of  the  SPS  users" 
may  not  be  met.  Three  years  of  effort  have  gone  into  ensuring  that 
the  “agreed  upon"  standard  processes  and  procedures  to  be  used  by  the 
Procurement  Community  in  the  future  are  identified  and  these 
standards  will  be  provided  to  the  DoD  Procurement  user  community 
through  the  SPS  solicitation  and  program. 

DLA  nonconcurs  that  "actual  costs  could  exceed  proposed  costs". 

Every  effort  has  been  taken  to  ensure  that  the  cost  projections  for 
the  SPS  acquisition  were  realistic  and  even  conservative  in  terms  of 
costs  for  potential  development  that  might  be  required.  Initial  cost 
proposals  from  the  SPS  offerors  appear  to  validate  this  position. 
Through  the  solicitation  and  evaluation  process  in  place  for  SPS, 
offerors  appear  to  have  and  continue  to  gain  a  good  understanding  of 
the  SPS  requirements  and  feel  confident  they  can  cost  it  on  a  fixed 
price  basis  in  their  best  and  final  offer(s).  In  the  final  analysis, 
the  audit  team  has  failed  to  credit  the  roles  of  the  DLA  DAISRC  and 
OSD  MAISRC  processes  in  providing  oversight  to  the  SPS  program. 

These  bodies  are  monitoring  the  SPS  program  in  accordance  with  the 
DoD  5000  series  directives.  As  such,  unanticipated  cost  growth,  if 
any,  would  be  quickly  identified  and  the  appropriate  decision  makers 
afforded  the  opportunity  to  decide  if  the  additional  costs  are 
justified  or  the  program  re-directed/terminated. 

DLA  concurs  with  the  finding  that  the  significance  of  the  SPS  Shared 
Data  Warehouse  has  not  been  stressed  within  existing  documentation  or 
project  plans. 
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Final  Report 
Reference 


Revised 
Page  15 


AUDIT  TITLE:  Allegations  to  the  Defense  Hotline  Concerning  the 
Standard  Procurement  System,  5RE-8019 

RECOMMENDATION  A. 2:  Recommend  that  the  Director,  Defense  Procurement 
Corporate  Information  Management  Systems  Center,  Defense  Logistics 
Agency : 

a.  Quantitatively  determine  the  most  cost  beneficial  deployment 
approach  and  if  applicable,  justify  deviations. 

b.  Include  in  the  Standard  Procurement  System  Program  Management 
Plan,  specific  plans  to  establish  the  Standard  Procurement  System 
Shared  Data  Warehouse.  That  description  should  include  both  known 
and  anticipated  risk  factors  and  associated  abatement  plans . 

DLA  COMMENTS:  Concur.  Recommendation  A. 2. a.  is  considered  an 
existing  requirement  under  the  DoD  5000  series  directives.  As  such, 
it  will  be  initially  accomplished  with  the  submittal  and  approval  of 
the  SPS  Milestone  II/IIIA  Economic  Analysis  (EA)  in  March  1997  for 
initial  deployment.  Updates  to  the  SPS  EA  documenting  each 
additional  deployment  recommendation  will  be  submitted  prior  to  those 
MAISRC  decisions.  Subsequent  SPS  milestone  decisions  are  not 
currently  scheduled,  but  are  anticipated  at  12-to-18  month  intervals 
after  the  Milestone  II/IIIA  decision. 

Inclusion  of  plans  for  the  Shared  Data  Warehouse  in  the  SPS  Program 
Management  Plan  will  be  completed  prior  to  the  SPS  Milestone  II/IIIA 
decision  in  March  1997  and  updated  as  appropriate  during  each 
additional  SPS  deployment  phase . 

DISPOSITION:  Ongoing.  ECD:  March  1997  and  subsequently  at  12-to-18 
month  intervals  until  the  SPS  Milestone  IV  decision  in  FY2001. 

DATE  BENEFITS  REALIZED;  Complete  by  FY  2001 

ACTION  OFFICER:  Capt .  E.J.  Case,  SC,  USN,  AQAC 

PSE  APPROVAL:  Gary  S.  Thurber,  Deputy,  Defense  Contract  Management 
Command 

COORDINATION;  Dave  Stumpf,  DDAI ,  767-6266 


DLA  APPROVAL: 


Major  Genera,!,  UcVv 
.Principal  Deputy  DircciM* 
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AUDIT  TITLE:  Allegations  to  the  Defense  Hotline  Concerning  the 
Standard  Procurement  System,  5RE-8019 

FINDING  B;  SPS  Testing  Strategy.  The  SPS  program  manager  has  not 
developed  adequate  developmental  and  operational  test  strategies  for 
the  SPS.  The  testing  strategy  is  inadequate  because: 

the  Procurement  Corporate  Information  Management  Council  has 
neither  provided  the  SPS  program  management  office  and  the  test 
community  with  user- validated  operational  requirements  nor  defined 
the  content  of  each  functional  increment, the  compressed  integrated 
program  schedule  does  not  allow  sufficient  time  to  ensure  the 
development  of  comprehensive  test  plans,  and  the  Council  has  not 
identified  requirements  to  the  program  manager  and  test  community  for 
the  Shared  Data  Warehouse  portion  of  the  SPS  program  to  enable  the 
development  of  a  testing  strategy. 

As  a  result,  the  SPS  program  may  not  meet  user  requirements. 

DLA  COMMENTS:  Partially  concur.  The  SPS  Council  has  chartered  a 
Working  Integrated  Product  Team  (WIPT)  to  develop  a  SPS  Operational 
Requirements  Document  (ORD) .  Those  portions  of  the  ORD  which  define 
operational  requirements  needed  by  the  test  community  are  the  highest 
priority  and  will  be  made  available  to  the  test  community  in  time  for 
completion  of  all  test  planning.  Prior  to  the  ORD  charter,  the  SPS 
PM  had  funded  and  supported  the  gathering  of  operational  performance 
requirements  for  testing  the  objective  system.  The  operational 
performance  requirements  along  with  their  respective  performance 
thresholds  and  objectives  were  gathered  from  users  at  nine  (9) 
representative  Service  and  Agency  procurement  legacy  sites.  The 
operational  performance  requirements  were  discussed  and  refined  at  an 
April  1996  SPS  Test  and  Evaluation  Working  Group  (TEWG)  meeting  by 
functional  users.  The  SPS  Operational  Test  Authority  (OTA)  was 
present  during  these  discussions  and  have  been  provided  a  copy  to 
support  their  test  planning.  The  refined  user  operational 
performance  requirements  were  staffed  by  the  Service  and  Agency 
representatives  of  the  SPS  Council .  These  user  validated 
operational  performance  requirements  will  be  included  in  a  Test  and 
Evaluation  Master  Plan  appendix  and  the  Acquisition  Program  Baseline. 
Validated  operational  performance  requirements  will  be  used  in 
support  of  testing  each  increment  upon  contract  award.  The  SPS  PM  is 
working  with  OSD  (DOT&E  &  DTSE&E)  in  refining  these  user  validated 
operational  performance  requirements  in  the  interest  of  facilitating 
ORD  development  for  SPS  under  the  new  DoDD  5000.2  guidance.  The  ORD 
is  planned  to  be  provided  by  15  Nov  96,  but  no  later  than  sixty  days 
prior  to  the  Operational  Test  Readiness  Review  (OTRR)  .  Both  the  ORD 
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and  the  operational  test  requirements  will  be  user  validated. 

The  OSD  testing  community  is  now  participating  in  all  SPS  test 
planning  in  accordance  with  DoD  5000  series  Integrated  Product  Team 
(IPT)  directions  and  has  voiced  no  dissatisfaction  with  the  tailored 
approach  and  time  lines  SPS  is  using  to  meet  DoD  AIS  test 
requirements.  The  OSD  Major  Automated  Information  Systems  Resource 
Council  (MAI SRC)  will  not  grant  deployment  approval  for  SPS  until  all 
required  testing  is  completed  as  certified  by  the  OSD  test  community. 

The  SPS  Source  Selection  Advisory  Council  directed  that  the  full  SPS 
requirement  not  be  segmented  into  functional  increments  in  the 
solicitation.  This  was  done  so  that  industry  could  propose  and 
obtain  evaluation  credit  for  the  most  functionality  that  their 
existing  commercial  product  has  or  which  is  currently  in  development . 
Each  offeror  is  being  afforded  the  opportunity  to  propose  what 
functionality  they  will  provide  in  up  to  four  releases,  but 
government /offeror  negotiated  contracts  will  determine  the  final 
functionality  to  be  provided  in  each.  In  this  manner,  at  contract 
award,  the  government  will  be  fully  aware  of  what  functionality  will 
be  available  in  each  release  in  order  to  plan  both  testing  and 
deployments.  As  such,  there  is  no  need  for  the  SPS  Council  to 
provide  functional  increments. 

The  SPS  Test  WIPT  feels  that  they  are  working  an  aggressive  testing 
schedule,  but  not  a  compressed  one  nor  one  which  has  insufficient 
test  planning  time  or  test  execution  time.  The  SPS  DEM/VAL  Test  Plan 
has  gone  through  multiple  iterations  and  reviews  by  the  test  site 
community,  and  the  next  version  will  be  ready  for  review  at  the  July 
10,  1996,  SPS  Test  WIPT.  The  Operational  Test  &  Evaluation  Plan 
(OTEP)  is  currently  in  development  by  the  Operational  Test  Authority 
(OTA)  at  JITC  Ft,  Huachuca.  Additional  T&E  Integrated  Product  Team 
(IPT)  meetings  are  scheduled  between  now  and  the  commencement  of 
DEM/VAL  testing,  which  include  participants  from  DOT&E,  DTSE&E,  DT 
and  OT  Test /evaluators ,  DLA  DAISRC/Test  Community,  and  NSA  (Security) 
to  discuss  the  Milestone  II/IIIA  TEMP  focusing  on  increment  1 
testing. 

Given  the  above,  the  stated  result  that  the  SPS  program  may  not  meet 
users  requirements  due  to  inadequate  testing  is  not  seen  as  a 
probable  outcome,  for  the  functionality  currently  defined  for  SPS. 
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AUDIT  TITLE;  Allegations  to  the  Defense  Hotline  Concerning  the 
Standard  Procurement  System,  5RE-8019 


RECOMMENDATION  B.2: 


Recommend  that  the  Director,  Defense  Procurement 
Corporate  Information  Management  Systems  Center, 
Defense  Logistics  Agency: 


a.  Establish  an  Integrated  Test  Program  Schedule  that  allows 
sufficient  time  for  detailed  test  planning  and  the  required  briefings 
and  review  by  the  Director  of  Operational  Test  and  Evaluation. 

b.  Develop  the  testing  strategy  for  shared  data  based  on  the 
requirements  definition  provided  by  the  Procurement  Corporate 
Information  Management  Council. 

DLA  COMMENTS:  Partially  concurs.  DLA  does  not  concur  with  the 
recommendation  that  the  SPS  Integrated  Test  Program  Schedule  (ITPS) 
has  insufficient  time  for  test  planning,  briefings  and  reviews  by 
Director  of  Operational  Test  and  Evaluation.  However,  the  SPS  Test 
IPT  will  be  afforded  an  opportunity  to  again  review  the  SPS  ITPS  and 
will  make  changes  indicated  as  necessary  by  the  IPT.  ECDs  31  August 
1996. 

DLA  concurs  that  the  SPS  Shared  Data  Warehouse  test  strategy  needs  to 
be  developed  and  integrated  into  the  overall  SPS  test  strategy. 

DISPOSITION:  Ongoing.  ECD:  April  1999,  at  the  SPS  Milestone  II/IIIC 
review  by  OSD 

ACTION  OFFICER:  Gary  Thurston,  AQAC,  767-6399 

PSE  APPROVAL:  Gary  S.  Thurber,  Deputy,  Defense  Contract  Management 
Command 

COORDINATION:  Dave  Stumpf,  DDAI ,  767-6266 


DLA  APPROVAL: 


MbC0F~^ 

M^or  General,  USA 
Rincipal  Deputy  Director 
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